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ABSTRACT 


This  report  provides  guidance  to  implement  the  Set  Based  Design  (SBD) 
methodology  into  the  Department  of  Defense  (DOD)  acquisition  framework.  Deferring 
requirements  and  design  decisions  is  the  essence  of  SBD,  which  in  turn  defers  cost 
commitments,  allowing  more  flexibility  to  management  than  traditional  design 
methodologies.  This  reduces  the  risk  for  cost  and  schedule  overruns,  both  of  which  are 
perennial  challenges  for  the  DOD.  This  report  identifies  the  original  SBD  principles  and 
characteristics  based  on  Toyota  Motor  Corporation’s  Set  Based  Concurrent  Engineering 
Model.  Additionally,  the  team  reviewed  DOD  case  studies  that  implemented  SBD.  The 
SBD  principles,  along  with  the  common  themes  from  the  case  studies,  are  then  analyzed, 
and  guidance  is  presented  for  implementing  SBD  into  the  Navy’s  2-pass/6-gate 
acquisition  governance  process  as  dictated  by  the  Secretary  of  the  Navy  acquisition 
instructions.  Recommendations  are  provided  on  the  system  factors,  such  as  program  type 
and  tool  infrastructure,  that  provide  a  good  fit  for  utilizing  SBD.  The  cost  and  schedule 
differences  between  SBD  and  a  typical  point-based  design  approach  are  discussed. 
Finally,  this  report  summarizes  the  findings  and  provides  program  managers  and  systems 
engineers  an  implementation  method  for  SBD  in  DOD  acquisition. 
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EXECUTIVE  SUMMARY 


Set  Based  Design  (SBD)  is  a  systems  engineering  methodology  that  explores 
design  combinations  via  a  systematic  elimination  of  infeasible  sets  until  reaching  a 
solution.  By  utilizing  concurrent  design  efforts  while  deferring  detailed  requirements 
until  they  are  fully  understood,  this  methodology  has  the  potential  to  replace  the  current 
point-based  design  structure  in  DOD  acquisition,  which  have  been  plagued  by  cost  and 
schedule  overruns.  Scope  creep  and  requirements  volatility  in  point-based  design 
implementations  often  result  in  major  rework,  a  major  contributor  to  the  cost  overruns 
and  delays  in  fielding  systems  to  the  warfighter.  Deferring  requirements  and  design 
decisions  is  the  essence  of  SBD,  which  in  turn  defers  cost  commitments  allowing  more 
flexibility  to  management,  reducing  the  risks  for  cost  and  schedule  overruns.  Although 
there  have  been  efforts  to  use  certain  aspects  of  SBD  in  acquisition,  it  has  not  been 
leveraged  to  the  maximum  extent,  nor  are  there  any  official  guidelines  or  instructions  for 
implementing  SBD.  The  objective  of  this  report  is  to  take  the  major  principles  and 
characteristics  of  SBD  and  provide  guidance  on  integrating  these  factors  into  DOD 
acquisitions.  Through  examination  of  industry  studies  and  DOD  instances  of  SBD,  this 
report  presents  recommendations  for  tailoring  the  acquisition  process  within  governing 
literature. 

Toyota  Motor  Corporation  employs  a  unique  design  approach  that  has  been 
dubbed  “Set  Based  Concurrent  Engineering,”  or  SBD.  Sobek  and  his  team  identified 
three  major  principles  of  SBD:  mapping  the  design  space,  integrating  by  intersection,  and 
establishing  feasibility  before  commitment  (Sobek  et  al.  1999).  Within  these  principles, 
SBD  is  further  broken  down  into  supporting  principles  to  provide  a  guide  for  the 
implementation  of  SBD.  Ghosh  and  Seering  also  examined  Toyota,  along  with  other 
instances  of  SBD,  and  published  a  list  of  the  seven  characteristics  of  set  based  thinking. 
The  characteristics  serve  as  a  “how  to”  guide  as  well  as  emphasizing  a  particular 
organizational  structure  that  enables  SBD  (Ghosh  and  Seering  2014). 
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The  characteristics  are: 

1.  Emphasis  on  Frequent  Low-Fidelity  Prototyping 

2.  Tolerance  for  Under-Defined  System  Specifications 

3.  More  Efficient  Communications  among  Subsystems 

4.  Emphasis  on  Documenting  Lessons  Learned 

5.  Support  for  Decentralized  Leadership  Structure  and  Distributed,  Non- 
Collocated  Teams 

6.  Supplier/Subsystem  Exploration  of  Optimality 

7.  Support  for  Flow-up  Knowledge  Creation 

This  report  considers  the  seven  characteristics  along  with  the  three  principles 
identified  above  which  form  the  foundation  of  SBD  for  the  implementation  guidance 
being  recommended. 

Keeping  these  principles  and  characteristics  in  mind,  examples  of  the  use  of  SBD 
in  DOD  system  acquisition  are  examined  to  determine  if  SBD  was  beneficial  and  if  SBD 
could  be  applied  to  government  acquisitions  successfully.  Each  of  these  systems  utilized 
Set  Based  Design  in  the  design  process.  Four  case  studies  are  reviewed  for  the  SBD 
impacts:  the  Ship  to  Shore  Connector,  the  Amphibious  Combat  Vehicle,  the  Small 
Surface  Combatant,  and  the  Large  Displacement  Unmanned  Underwater  Vehicle.  An 
additional  section  in  this  chapter  covers  a  study  conducted  by  Naval  Surface  Warfare 
Center  Carderock  Division,  in  which  the  Carderock  employees  examined  the  differences 
between  point-based  design  and  SBD  and  their  impacts  on  a  ship  design,  resulting  in 
several  takeaways.  In  all  cases,  design  development  was  not  limited  to  a  single  solution, 
and  by  delaying  design  decisions  until  realizing  and  understanding  trade-offs,  a  longer 
period  for  stakeholder  influence  and  feedback  resulted.  However,  there  were  some 
drawbacks.  In  some  cases,  there  were  higher  initial  costs  and  commitment  of  resources 
upfront  to  conduct  the  SBD  analysis  than  in  a  point-based  design  implementation.  There 
was  also  a  lack  of  education  and  experience  in  the  execution  and  implementation  of  SBD. 
Overall,  the  use  of  SBD  in  the  case  studies  has  proven  beneficial,  to  include  the  case 
study  takeaways  and  their  use  of  the  three  principles  and  seven  characteristics  of  SBD. 


The  high-level  governing  document  for  DOD  acquisition  is  the  Operation  of  the 
Defense  Acquisition  System  or  DODI  5000.02.  This  instruction  provides  guidance  to  the 
services  for  interpretation  and  implementation  of  acquisition  processes.  For  the  Navy,  the 
Secretary  of  the  Navy  has  signed  his  own  instruction,  Department  of  the  Navy 
Implementation  and  Operation  of  the  Defense  Acquisition  System  and  the  Joint 
Capabilities  Integration  and  Development  System  or  the  SECNAVINST  5000.02E.  This 
lays  out  a  system  for  acquisition  within  the  Navy  and  Marine  Corps,  which  has  a  2-Pass/ 
6-Gate  process  for  meeting  the  required  goals  per  Milestones  A,  B,  and  C  within  the 
DODI  5000.02.  After  analyzing  the  Navy’s  acquisition  process,  two  guiding  strategies  for 
employing  SBD  within  the  current  framework  emerge.  The  first  scenario  is  to  incorporate 
the  use  of  SBD  from  pre-Milestone  A,  or  the  Material  Development  Decision,  until 
Milestone  B.  This  strategy  would  result  in  a  Request  for  Proposal  (REP)  to  the  defense 
industry  to  complete  the  detailed  design  of  the  system.  The  second  option  would  be  to 
implement  SBD  from  the  same  origin  as  the  first  scenario,  but  continue  the  SBD  efforts 
until  meeting  the  entrance  criteria  for  Milestone  C.  This  would  result  in  a  Technical  Data 
Package  to  enable  a  production  REP  to  a  defense  industry  vendor,  or  to  produce  Low 
Rate  Initial  Production  items  for  testing  and  other  Milestone  C  entrance  activities.  The 
focus  of  the  implementation  is  to  look  for  the  system  design  attributes  that  have  the 
largest  impact  on  design  instead  of  narrowing  down  to  the  requirements  right  away. 
Acquisition  programs  would  then  take  the  high-level  capabilities  and  group  them  based 
on  mission  or  concept  of  operations.  From  these  groupings,  the  design  further  narrows  as 
infeasible  sets  are  no  longer  viable,  leaving  an  initial  product  baseline  at  the  Critical 
Design  Review. 

The  report  also  presents  program  factors  for  examination  prior  to  adopting  the 
SBD  methodology.  The  team  recommends  instances  when  to  use  SBD  based  on  the 
acquisition  model  utilized  and  the  capability  of  the  tools  in-house  to  analyze  all  the  sets 
of  data  to  narrow  the  design.  Cost  and  schedule  risk  factors  are  big  drivers  for  the  use  of 
SBD.  The  upfront  analysis  conducted  increases  cost  and  schedule  requirements  early  in 
the  program  life  cycle,  therefore  changes  in  program  development  cost  and  schedule  are 
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necessary  if  one  pursues  SBD.  Utilizing  SBD  should  minimize  rework  and  thus,  lower 
the  risk  of  both  cost  and  schedule  overruns. 

This  report  concludes  that  the  strict  construct  of  DOD  acquisition  does  indeed 
support  the  SBD  principles.  Utilizing  the  lessons  learned  from  Toyota,  the  derived 
principles  and  characteristics,  and  past  uses  of  SBD  in  the  DOD,  guidance  has  been 
provided  for  consideration  when  implementing  SBD  as  a  systems  engineering 
methodology. 
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I. 


INTRODUCTION 


A.  BACKGROUND 

In  the  ever-changing  fiscal  and  geopolitical  environment,  recent  defense 
acquisition  reform  has  been  aimed  to  reduce  system  development  time  and  cost.  One 
methodology  examined  as  part  of  reform  efforts  is  Set  Based  Design  (SBD).  Set  Based 
Design  is  a  systems  engineering  methodology  that  explores  design  combinations  via  a 
systematic  elimination  of  infeasible  sets  until  a  solution  emerges.  The  purpose  of  this 
study  is  to  examine  SBD  and  analyze  its  potential  application  to  defense  acquisition  to 
grow  military  technologies  and  apply  them  to  systems  at  a  rate  greater  than  that  of  our 
potential  adversaries.  The  practice  of  SBD  has  merit  both  in  the  civilian  and  military 
sectors,  yet  has  not  been  formally  incorporated  into  the  Department  of  Defense  (DOD) 
acquisition  life  cycle.  The  study  establishes  recommendations  for  applying  SBD 
methodologies,  considering  the  potential  advantages  and  risks. 

B.  PROBLEM  STATEMENT 

The  United  States  needs  to  maintain  maritime  superiority  as  near  peer  threats 
expand  their  maritime  capability.  However,  according  to  a  2013  Government 
Accountability  Office  (GAO)  study,  “The  cost  growth  of  DOD’s  2012  portfolio  of 
weapon  systems  [was]  about  $411  billion  and  schedule  delays  average  more  than  2 
years”  (1).  One  of  the  GAO  recommended  areas  of  improvement  is,  “identifying 
significant  risks  up  front  and  resourcing  them”  (2013,  12). 

Many  weapon  system  development  efforts  are  experiencing  cost  growth  and 
schedule  delays  due  to  requirements  volatility.  The  traditional  method  of  point-based 
design  (PBD)  selects  one  alternative  for  development  at  the  onset  of  the  program  (Ghosh 
and  Seering  2014).  A  problem  with  PBD  is  that  engineers  do  not  fully  understand 
requirements  at  this  point  and  changes  to  requirements  yield  rework:  “The  typical  goal 
[of  traditional  systems  engineering]  is  to  get  the  requirements  and  specifications  nailed 
down  as  early  as  possible”  (Kennedy  et  al.  2013,  11).  Kennedy  continues  by  describing 
the  risk  of  nailing  down  requirements  too  early:  “However,  tightly  specifying 
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requirements  early  in  the  project  means  that  some  of  the  critical  decisions  are  made  very 
early.  If  they  are  made  with  too  little  knowledge  of  what  customers  really  want  or  what  is 
technically  possible,  then  rework  is  inevitable”  (2013,  11).  Rework  equates  to  cost 
growth  and  schedule  overruns. 

The  SBD  methodology  is  one  potential  solution  to  this  problem,  as  it  has  been 
proven  in  the  commercial  market.  SBD  is  a  design  methodology  used  to  expand  the 
design  space,  including  ample  design  possibilities,  while  delaying  critical  design 
decisions  until  the  right  time,  to  narrow  the  set  of  designs  systematically  by  identifying 
and  eliminating  infeasible  solutions,  while  integrating  the  intersections  of  feasible 
designs  (Sobek  et  al.  1999).  Applying  SBD  effectively  means  delaying  critical  decisions 
until  a  better  understanding  of  the  problem  arises,  potentially  resulting  in  a  timely  and 
cost-effective  identification  of  the  right  solution. 

At  this  time,  no  activity  has  fully  incorporated  SBD,  and  there  exists  no  DOD- 
wide  and  no  Navy-wide  guidance  on  how  program  managers  can  apply  it  to  leverage  the 
potential  cost  savings  and  schedule  benefits  through  the  reduction  of  rework.  The  purpose 
of  this  report  is  to  provide  such  guidance  for  the  acquisition  community. 

C.  PROJECT  SCOPE 

This  report  considers  how  the  DOD  acquisition  process  can  leverage  the  SBD 
methodology  to  deliver  more  affordable  systems  to  the  fleet  faster.  The  research  focuses 
on  defining  SBD  and  its  core  principles,  as  well  as  the  understanding  previous 
applications  of  SBD  in  both  industry  and  the  DOD,  to  gain  insight  into  appropriate  uses 
and  implementation  processes.  Primary  source  documentation  from  several  DOD 
programs,  including  the  Ship  to  Shore  Connector  (SSC),  the  Amphibious  Combat 
Vehicle  (ACV),  the  Small  Surface  Combatant,  and  the  Large  Displacement  Unmanned 
Underwater  Vehicle  (LDUUV),  was  used  to  develop  case  studies  to  better  understand 
how  SBD  has  been  used  in  the  past  and  determine  how  best  to  use  it  in  the  future.  The 
objective,  therefore,  is  to  determine  how  to  tailor  an  acquisition  strategy  to  incorporate 
elements  of  SBD  to  manage  cost  growth  and  scheduling  delays  due  to  changing 
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requirements  and  the  resultant  design  volatility.  Additionally,  we  will  provide  guidance 
on  what  aspects  of  the  acquisition  environment  would  allow  for  such  an  approach. 

The  project  reports  on  the  following: 

1.  A  description  of  the  evolution  of  SBD,  and  its  major  principles  and 
characteristics 

2.  An  exploration  of  various  implementations  of  SBD  in  the  civilian  and 
military  sectors  alike 

3.  A  brief  description  of  the  governing  documents  for  DOD  acquisition 

4.  Identification  of  system  types  that  make  good  candidates  for  the 
application  of  SBD 

5.  Identification  of  system  types  for  which  SBD  would  not  be  recommended 

6.  Recommended  implementation  practices  and  processes,  within  the 
governing  instructions,  for  the  use  of  SBD  into  the  DOD  acquisition  life 
cycle 

This  project  sought  to  answer  the  following  questions: 

1.  What  is  SBD  and  how  can  it  benefit  defense  acquisition? 

2.  What  factors  make  a  program  a  good  candidate  for  employing  a  SBD 
approach  in  defense  acquisition? 

3.  What  effect  does  SBD  have  on  overall  system  costs  and  risks  in  support  of 
defense  acquisition?  Are  the  potential  benefits  worth  it? 

4.  What  instructions  and  processes  would  have  to  be  tailored  or  revised  to 
facilitate  Programs  of  Record  (PORs)  to  use  SBD  in  their  development 
activities? 


D.  SYSTEMS  ENGINEERING  PROCESS 

The  team  employed  a  tailored  waterfall  process  model  in  order  to  explore  SBD 
applications  in  the  support  of  defense  acquisition  and  PORs.  Figure  1  shows  the  roadmap, 
from  post  problem  definition  to  project  report. 
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Figure  1.  Capstone  Engineering  Process. 


1.  SBD  Analysis 

The  team  reviewed  the  SBD  literature  to  define  properly  the  SBD  principles  and 
characteristics  utilized  for  the  study.  The  history  of  SBD  and  the  analyses  of  several  case 
studies  help  determine  the  capabilities  and  limitations  of  SBD  to  better  prepare  for  its 
introduction  into  the  acquisition  life  cycle.  This  section  was  iterative  in  nature,  based  on 
lessons  learned  from  the  various  case  studies. 

2.  DOD  SBD  Case  Studies 

The  team  examined  current  applications  of  SBD  in  DOD  acquisition.  This  process 
analyzed  primary  source  SBD  documentation  in  several  defense  programs,  providing  a 
history  of  the  projects  and  programs,  the  issues  and  constraints,  how  SBD  was  utilized, 
and  the  successes  or  failures  of  utilizing  SBD.  By  applying  what  we  learned  about  SBD, 
we  developed  case  studies  to  identify  the  potential  benefits  and  programmatic  risks  of 
applying  SBD  to  the  acquisition  life  cycle. 

3.  Implementation  of  SBD  into  the  Acquisition  Life  Cycle 

The  team  studied  the  SBD  principles  and  lessons  learned  defined  from  the 
previous  phases  and  determined  where  the  acquisition  life  cycle  should  adapt  in  order  to 
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execute  SBD.  Feedback  from  the  implementation  changed  the  initial  thoughts  of  the  SBD 
Analysis  and  the  principles  originally  defined.  The  focus  centered  on  the  major  system 
engineering  functions  such  as  the  analysis  of  alternatives  (AoA),  System  Engineering 
Technical  Reviews,  and  the  prototyping  test  strategy,  as  found  in  the  Operation  of  the 
Defense  Acquisition  System  (DODI  5000.02)  and  Department  of  the  Navy 
Implementation  and  Operation  of  the  Defense  Acquisition  System  and  the  Joint 
Capabilities  Integration  and  Development  System  (SECNAVINST  5000. 2E).  The  team 
formulated  a  new  Defense  Acquisition  Program  model  as  well  as  requirements  for  the 
different  technical  reviews  and  decision  points  along  the  acquisition  life  cycle. 

4.  Conclusions 

The  team  explored  the  findings  and  summarized  the  SBD  implementation  and 
lessons  learned  from  the  case  studies  reviewed.  Based  on  feedback  from  each  stage,  the 
team  was  able  to  provide  guidance  for  SBD  implementation  into  the  acquisition  life 
cycle. 
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II.  REVIEW  OF  SET  BASED  DESIGN 


This  chapter  explores  SBD  to  set  a  solid  foundation  for  our  analysis.  We  present  a 
brief  history  of  SBD,  starting  with  the  origins  from  Toyota  and  its  concurrent  engineering 
concepts.  Next,  we  present  the  high-level  SBD  principles  and  their  supporting  elements 
as  determined  by  Ward  and  his  coauthors  in  their  1995  work  “The  Second  Toyota 
Paradox:  How  Delaying  Decisions  Can  Make  Better  Cars  Faster.”  The  decade  following 
this  work  motivated  additional  research  of  the  subject,  one  of  which  was  “Set-Based 
Thinking  in  the  Engineering  Design  Community  and  Beyond”  by  Ghosh  and  Seering  in 
2014.  Ghosh  and  Seering  reviewed  Ward  and  other  authors  to  provide  their 
understanding  of  SBD  methodology,  consolidating  the  three  principles  presented  by 
Ward  et  al.  to  two  and  naming  seven  characteristics  of  set-based  product  development. 
Finally,  a  look  into  the  Wright  brother’s  development  of  the  first  manned,  machine 
powered  flight  provides  a  concrete  example  of  the  potential  cost  savings  and  schedule 
benefits.  Based  on  these  works,  and  others,  SBD  was  studied  to  provide  a  better 
understanding  of  its  uses  and  potential  implementation  into  the  DOD  acquisition  life 
cycle. 

A.  HISTORY 

Traditionally,  the  design  approach  to  naval  systems  employed  the  use  of  Point- 
Based  Design  (PBD)  to  develop  products.  The  execution  of  PBD  is  either  employed  in 
series  or  concurrently  (Sobek  et  al.  1999).  In  serial  PBD,  a  finalized  component  of  the 
design  passes  to  the  designers  of  the  next  component.  In  concurrent  PBD,  designers 
choose  an  initial  best  solution  approach,  then  iterate  with  increasing  detail,  incorporating 
feedback  from  other  designers  until  the  final  design  emerges.  Figure  2  shows  these  two 
PBD  approaches  as  practiced  in  an  automobile  design  domain.  The  PBD  serial 
engineering  approach  conducts  engineering  as  a  series  of  functions  with  minimal 
feedback  loops.  Before  moving  on  to  the  next  step,  each  previous  function  must  be 
complete.  The  PBD  concurrent  engineering  approach  tries  to  conduct  parallel  processing 
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of  the  functions  to  obtain  feedback  earlier.  However,  both  PBD  approaches  still  require 
early  design  decisions  with  several  stages  of  iteration  on  one  solution. 


Traditional  Point-Based  Approach  Examples: 


Point-Based  Serial  Engineering 

- ►  - ►  - ►  - ► 

Styling  Marketing  Body  Chassis  Manufacturing 


Point-Based  Concurrent  Engineering  (Styling  Example) 


Design  Analyze 

Solution  - ►  and 

(Styling)  Critique 

q 


Modify 


Marketing 


Body 


Chassis 

Manufacturing 


Figure  2.  Point-Based  Design  Approaches.  Adapted  from  Sobek  et  al.  (1998,  69). 

In  naval  systems,  this  iterative  cycle  generally  initiates  after  one  alternative  from 
the  AoA  phase  is  chosen  and  pursued  through  preliminary  design,  critical  design, 
developmental  and  operational  test,  full  rate  production,  sustainment,  and  disposal, 
depicted  as  the  design  spiral  shown  in  Figure  3.  The  classical  design  spiral  follows  the 
PBD  serial  approach  where  the  satisfied  constraints  emerge  from  the  consideration  of 
each  design  requirement  in  sequence  (Singer  et  al.  2009). 
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Figure  3.  Classical  Design  Spiral.  Source:  Evans  (1959,  692). 


Efforts  to  speed  up  this  iterative  cycle  to  obtain  the  final  design  sooner  at  auto 
manufacturing  companies  in  the  United  States  have  mostly  focused  on  organizational 
changes,  including  the  use  of  cross-functional,  often  collocated,  teams  to  increase  the 
speed  and  effectiveness  of  communications  (Ward  et  al.  1995). 

However,  as  systems  have  become  more  complex  with  an  increasing  need  to 

produce  effective  designs  more  efficiently,  the  traditional  approach  to  design,  which 

narrows  and  fixes  a  design  early,  is  being  reconsidered  (Kennedy  et  al.  2013).  The  old 

PBD  approach  leaves  little  ability  to  refine  specifications  later  in  the  systems  engineering 

process.  Too  often  changes  are  being  requested  late  in  the  design  effort,  when 
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requirements  are  better  understood  (Kennedy  et  al.  2013).  The  relative  inflexibility  of  the 
PBD  approach  results  in  rework,  as  designers  revert  to  the  early  design  stages  to  select  a 
new  design,  which  fulfills  the  developing  requirements  (Kennedy  et  al.  2013).  The 
alternative  to  staring  over  is  to  accept  less  effective  designs,  which  is  likely  due  to 
schedule  and  fiscal  constraints;  neither  option  is  ideal. 

SBD  allows  the  design  discovery  for  multiple  efforts  to  take  place,  earlier  in  the 
design  phase,  before  deciding  upon  detailed,  finalized  alternatives.  More  specifically, 
SBD  allows  multiple  design  options  to  remain  viable  and  allows  for  feedback  and 
influence  from  stakeholders  throughout  the  design  process.  In  doing  so,  requirements  are 
understood  better  prior  to  making  finalized  decisions,  and  the  final  products  better  fulfill 
stakeholders’  needs  (Singer  et  al.  2009). 

As  early  as  1995,  Toyota  Motor  Corporation  was  one  of  the  first  to  implement 
SBD  successfully,  resulting  in  the  company  becoming  a  leading  competitor  in  the 
automotive  industry  (Ward  et  al.  1995).  Toyota’s  implementation  of  SBD  took  the  form 
of  what  they  call  Set  Based  Concurrent  Engineering  (SBCE).  The  “Toyota  Model”  is 
steeped  in  delayed  decisions,  ambiguous  communication,  and  the  pursuit  of  an  “excessive 
number  of  prototypes,”  which  helps  Toyota  to  design  better  cars  “faster  and  cheaper” 
(Ward  et  al.  1995,  44). 

The  main  features  of  Toyota’s  design  process,  according  to  Singer  et  al.  (2009) 
include: 

1.  Broad  sets  of  design  parameters  [being]  defined  to  allow  concurrent 
design  to  begin 

2.  Sets  [being]  kept  open  longer  than  typical,  to  more  fully  define  tradeoff 
information 

3.  The  sets  [being]  gradually  narrowed  until  a  more  globally  optimum 
solution  is  revealed  and  refined 

4.  As  the  sets  narrow,  the  level  of  detail  (or  design  fidelity)  increases 

As  demonstrated  in  Figure  4,  each  of  the  design  aspects,  including  the  marketing 
concept,  styling,  product  design,  components,  and  manufacturing  system  design,  are  kept 
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in  work  until  an  option  is  no  longer  viable  and  is  eliminated,  reducing  the  number  of 
design  options  (Ward  et  al.  1995). 


Figure  4.  Parallel  Set  Narrowing  Process  Sketched  by  Toyota  Manager.  Source: 

Ward  et  al.  (1995,  49). 

This  narrowing  ultimately  results  in  the  optimum  design  solution,  which  most 
accurately  reflects  the  requirements  needs  of  the  stakeholders.  The  chapter  lays  out 
Toyota’s  design  process  and  principles  later  on. 

By  delaying  decisions,  SBD,  in  practice,  also  allows  Toyota  and  other  SBD  users 
to  delay  the  commitment  of  costs.  In  general  terms,  by  locking  in  the  design  early  on  with 
little  understanding  of  the  system,  difficulty  arises  in  the  ability  to  influence  that  design 
without  negative  impacts  to  cost  and  schedule.  This  results  in  significant  schedule  delays 
and  cost  overruns,  should  the  desire  or  need  to  make  design  changes  be  present.  A  study 
conducted  in  1989,  by  a  U.S.  DOD  Technology  Assessment  Team,  “show[s]  that  seventy 
to  eighty  percent  or  more  of  the  projected  life  cycle  costs  are  built  in  at  the  planning  and 
design  stages”  (Neel  1991,  11).  Figure  5  corroborates  the  team’s  findings  by  showing 
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how  a  disproportionate  amount  of  life  cycle  costs  are  committed  by  the  design  activities 
early  on  in  the  development  phase  of  PBD  (Singer  et  al.  2009).  What  Toyota  strives  to  do 
is  make  their  committed  costs  more  closely  match  the  depicted  incurred  costs:  “SBD 
strives  to  reduce  the  Committed  Costs  to  more  closely  follow  the  Incurred  Costs”  (Singer 
et  al.  2009,  11).  This  technique  is  a  risk  mitigation  for  the  overall  budget. 


Percent  of  Development  Complete 

Figure  5.  Depiction  Rate  of  Life-Cycle  Cost  Commitment  vs.  Percent  of 
Development  Complete.  Source:  Singer  et  al.  (2009,  1 1). 

Due  to  the  realities  of  PBD,  much  of  these  committed  costs  are  the  result  of  early 
design  decisions,  made  with  an  insufficient  level  of  understanding.  Cost  commitments 
and  design  decisions  change,  most  often,  only  through  the  application  of  additional  cost 
and  time.  It  is  for  these  reasons  SBD  practices  and  principles  are  superior  to  and 
revolutionize  the  traditional  PBD  method. 
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In  order  to  more  closely  match  committed  costs  to  incurred  costs,  Toyota  incurs 
additional  development  costs  up  front  as  a  result  of  keeping  alternative  designs  alive,  to 
ensure  they  commit  to  the  best  overall  design  (Sobek  et  al.  1999).  Compared  to  U.S.  auto 
manufacturers,  Toyota  uses  a  significantly  larger  number  of  designs  on  the  drawing  board 
and  at  any  given  time  they  may  have,  “anywhere  from  five  to  twenty  different  styling 
alternatives”  (Sobek  et  al.  1999,  25).  This  number  of  alternatives  was  excessive,  and, 
until  Toyota  used  its  SBCE  approach,  was  previously  unheard  of  in  the  car  industry.  They 
have  proven  that  the  extra  upfront  engineering  investment  pays  off,  as  they  consistently 
occupy  spots  in  the  J.D.  Power  Top  Ten  for  initial  quality  and  time  from  program 
initiation  to  production.  On  average,  Toyota  was  at  the  production  phase  in  27  months, 
versus  29  months  at  Nissan  and  37  months  at  Chrysler  (Ward  et  al.  1995).  This  is 
accomplished  through  the  narrowing  technique  resulting  in  a  robust  design,  requiring 
very  little  rework,  more  closely  meeting  stakeholders’  requirements.  The  final  products 
proved  to  be  industry  successes. 

Toyota’s  success  with  SBD  caught  the  interest  of  industry  and  government 
acquisition  professionals,  so  much  so  that  it  inspired  the  Navy  to  begin  considering 
alternatives  to  the  traditional  PBD  acquisition  process.  The  first  use  of  SBD  in  Navy 
acquisitions  took  place  in  2007  when  Vice  Admiral  Paul  Sullivan,  Commander  of  Naval 
Sea  Systems  Command  (NAVSEA),  “agreed  to  allow  the  Ship  to  Shore  Connector  (SSC) 
Program  to  begin  a  government-led  preliminary  design  (PD)  and  contract  design  (CD)” 
utilizing  SBD  (Buckley  2011,  79).  In  doing  so,  the  SSC  Design  Team  contracted  Dr. 
David  Singer,  University  of  Michigan,  to  assist  in  the  implementation.  “Dr.  Singer  had 
conducted  extensive  research  on  the  use  of  SBD  for  ship  design”  (Buckley  et  al.  2011, 
80). 

On  February  4,  2008,  Admiral  Paul  Sullivan  sent  a  memo  to  his  workforce 
encouraging  the  application  of  SBD  in  the  acquisition  process  of  shipbuilding  (Singer  et 
al.  2009).  Following  the  use  of  SBD  for  the  SSC,  SBD  proves  to  be  a  feasible  alternative 
to  traditional  PBD  acquisition  in  more  recent  systems  acquisition  programs.  In  the  same 
year,  the  Secretary  of  the  Navy  (SECNAV)  implemented  a  “2-Pass/6-Gate”  process 
ensuring  stakeholders  are  involved  in  the  acquisition  decision  process  from  development 
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of  the  Initial  Capabilities  Document  (ICD)  through  design  and  construction,  in  other 
words,  between  AoA  and  Pre-Preliminary  Design  which  follows  Milestone  A. 

B.  SET  BASED  DESIGN  PRINCIPLES 

According  to  Dr.  Raudberget,  of  Jonkoping  University  Sweden,  SBD  or  SBCE,  as 
he  calls  it,  has  a  different  decision  methodology  from  the  traditional  PBD.  Raudberget 
says,  “the  SBCE  decision  process  is  based  on  a  rejection  of  the  least  suitable 
solutions... SBCE  carries  forward  all  implementations  that  cannot  yet  be  eliminated” 
(Raudberget  2010,  687).  Through  the  rejection  of  the  less  desirable  solutions,  the 
designers  methodically  narrow  the  remaining  set  of  solutions.  This  narrowing  of 
alternative  solution  sets  results  in  the  best  and  final  solution  becoming  evident.  To  do 
this,  Sobek,  Ward,  and  Liker  suggest  three  SBD  principles  based  on  their  research  of  the 
Toyota  Motor  development  process.  These  three  principles  are  Map  the  Design  Space, 
Integrate  by  Intersection,  and  Establish  Feasibility  before  Commitment  (Sobek  et  al. 
1999).  Based  on  Sobek  and  his  colleagues’  studies  of  Toyota,  they  decomposed  each 
SBD  principle  into  three  supporting  elements,  resulting  in  the  following  list: 

1 .  Map  the  Design  Space 

a.  Define  Feasible  Regions 

b.  Explore  Trade-Off  by  Designing  Multiple  Alternatives 

c.  Communicate  Sets  of  Possibilities 

2.  Integrate  by  Intersection 

a.  Look  for  the  Intersection  of  Feasible  Sets 

b.  Impose  Minimum  Constraint 

c.  Seek  Conceptual  Robustness 

3.  Establish  Feasibility  before  Commitment 

a.  Narrow  Sets  Gradually  while  Increasing  Detail 

b.  Stay  within  Sets  Once  Committed 

c.  Control  by  Managing  Uncertainty  at  Process  Gates 
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1.  Map  the  Design  Space 

The  first  principle,  Map  the  Design  Space,  has  three  elements:  Define  Feasible 
Regions,  Explore  Trade-Off  by  Designing  Multiple  Alternatives,  and  Communicate  Sets 
of  Possibilities  (Sobek  et  al.  1999).  Sobek  et  al.  observed  distinct  differences  at  Toyota 
versus  their  U.S.  automaker  counterparts.  Toyota  engineers  explore  and  communicate 
numerous  alternatives  between  their  engineering  divisions  to  gain  a  “thorough 
understanding  of  a  set  of  possibilities”  (Sobek  et  al.  1999,  73).  Raudberget  describes  a  set 
as  “a  palette  of  different  solutions  to  a  specific  function  or  problem  and  can  be  seen  as  a 
family  of  design  proposals”  (2010,  687).  To  determine  these  sets,  the  feasible  region  must 
be  defined  (Sobek  et  al.  1999). 

According  to  Sobek  et  al.  (1999),  to  Define  the  Feasible  Region,  each  functional 
department  at  Toyota  determines,  in  parallel,  the  design  constraints  or  “what  cannot  be 
done  or  should  not  be  done”  (73).  These  constraints  are  documented  in  the  “engineering 
checklists  (or  design  standards)”  maintained  by  every  engineering  functional  team  on  the 
project  (Sobek  et  al.  1999,  73).  These  are  not  requirements  or  specifications  but 
guidelines  based  on  knowledge  and  experience  of  the  details  of  numerous  constraints 
such  as  functionality,  reliability,  manufacturability,  government  regulation.  The  space 
within  these  constraints  is  therefore  the  feasible  region. 

Once  the  feasible  region  is  defined,  Sobek  et  al.  (1999)  describe  how  the 
functional  teams  Explore  the  Trade-Offs  of  multiple  design  alternatives.  They  state  that 
to  do  this,  Toyota  and  its  suppliers  simulate  and/or  prototype  system  and  subsystem 
alternatives.  Single  data  points  are  much  less  useful  than  curves,  so  as  much  as  possible, 
establishing  relationships  between  parameters,  via  trade-off  curves,  enables  a  successful 
analysis.  At  Toyota,  they  value  the  reassurance  of  the  best-chosen  solution  more  than  the 
inefficiency  or  cost  that  may  have  resulted  from  finding  that  solution,  according  to  Sobek 
et  al.  For  example,  one  Toyota  exhaust  supplier  develops  approximately  10  to  20  exhaust 
prototypes  for  each  new  Toyota  car  program  (in  an  extreme  case  the  supplier  made  50 
prototypes  for  one  new  car  program).  Sobek  et  al.  continue,  explaining  that  the  exhaust 
supplier  uses  an  engine  supplied  by  Toyota  with  the  prototype  exhaust  systems  to 

produce  trade-off  curves  for  several  variables,  such  as  backpressure  versus  noise 
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reduction.  In  contrast,  at  Chrysler,  the  design  team  iterates  on  only  the  best  idea.  For 
instance,  the  Chrysler  body-engineering  group  only  considers  the  most  likely  design  and 
does  not  begin  detailed  design  until  the  styling  is  set.  Toyota’s  body-engineering  team 
develops  two,  three,  or  more  of  some  body  subsystems  to  determine  the  impacts  of  body 
styling  before  the  final  styling  is  determined  (Sobek  et  al.  1999). 

Communicating  Sets  of  Possibilities  is  the  third  element  of  mapping  the  design 
space  (Sobek  et  al.  1999).  Communicating  all  sets  of  possibilities  is  critical  in  order  to 
determine  the  best  solution  for  the  overall  system.  The  exchange  of  the  sets  enables 
discovery  of  the  best  global  solution,  not  just  the  best  solution  from  a  single  functional 
engineering  group’s  point  of  view  (Sobek  et  al.  1999). 

Toyota  engineers  communicate  a  variety  of  different  information  types  and  data. 
These  communications  are  explicit  and  not  casual  (Sobek  et  al.  1999).  One  of  the 
standardized  forms  utilized  by  Toyota  engineers  is  a  design  matrix  that  simply  compares 
alternatives  on  one  axis  to  design  criteria  on  the  other,  the  comparison  can  either  be 
qualitative  or  quantitative  (Sobek  et  al.  1999).  Table  1  depicts  a  qualitative  comparison 
showing  the  range  of  performance  or  value  for  a  particular  function  or  attribute  across 
various  alternatives.  Other  information  exchanged  beyond  discrete  alternatives,  includes 
lists  of  ideas,  drawings,  models,  subsystem  constraints  and  interfaces,  trade-off  curves, 
performance  charts,  and  estimates  (Sobek  et  al.  1999). 


Table  1.  Example  Alternative  Matrix.  Adapted  from  Sobek  et  al. 

(1999,  76). 


Matrix  of  Communicating  Alternatives 

Alternative 

Function  1 

Function  2 

Cost 

Space 

Etc. 

X 

® 

® 

♦ 

□ 

Y 

♦ 

□ 

O 

® 

Z 

® 

♦ 

♦ 

O 

O- Excellent  ©-Acceptable  ♦-Marginal  Q  Unacceptable 
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2.  Integrate  by  Intersection 

The  second  SBD  principle  is  Integrate  by  Intersection,  which  contains  three 
elements:  Looking  for  Intersections  of  Feasible  Sets,  Imposing  Minimum  Constraint,  and 
Seeking  Conceptual  Robustness  (Sobek  et  al.  1999).  An  intersection,  defined  by  Sobek  et 
al.,  is  a  solution  that  is  satisfactory  to  all  (1999).  The  purpose  of  identifying  intersections 
is  to  determine  which  alternatives  are  feasible,  so  they  can  eliminate  the  infeasible.  To  do 
this,  Toyota  engineers  focus  on  a  solution  that  is  best  for  the  whole  system.  That  is,  each 
set  considers  all  subsystems  in  that  set,  while  making  trades  based  on  the  collection  of  the 
complete  set,  not  the  optimization  of  each  individual  subsystem.  In  other  words,  it  is  a 
more  holistic  approach,  and  one  functional  area  may  give  way  in  the  interest  of  the 
whole.  Even  if  such  trades  result  in  degraded  performance  in  a  given  functional  area,  it  is 
important  that  the  system  is  maximized  overall  (Sobek  et  al.  1999).  To  do  this,  Toyota 
developers  invoke  the  principle  of  “Nemawashi,”  which  translates  into  doing  the 
“groundwork,”  to  include  meeting  and  communicating  with  all  stakeholders  (Sobek  et  al. 
1999,  77).  Through  this  stakeholder  interaction,  they  ensure  that  the  developing 
requirements  and  technology  are  in  fact  contributing  to  a  better  overall  design.  The 
engineering  efforts  for  the  various  attributes  are  concurrent  and  the  overall  solution 
concept  is  not  set  in  stone;  identifying  the  solutions  that  occur  at  the  intersections  of 
feasible  sets  is  a  crucial  filter  and  reaffirms  that  the  best  overall  system  solution  emerges 
within  the  remaining  sets  under  consideration. 

The  second  element  of  Imposing  Minimum  Constraint,  as  described  by  Sobek  and 
his  colleagues  (1999),  maintains  design  flexibility  to  make  advantageous  adjustments 
during  integration  as  design  exploration  continues.  The  balance  is  to  pose  “just  enough 
constraint”  that  each  of  the  subsystems  operates  correctly,  but  no  more,  thereby  enabling 
final  design  optimization  (78).  The  practice  manifests  advantages  in  multiple  places;  one 
example  is  the  interaction  between  the  final  computer  aided  design  (CAD)  drawings  and 
the  manufacturing  die  determination.  Toyota  initially  sends  CAD  data  to  the 
manufacturing  engineering  group  with  no  tolerances.  The  manufacturing  group  designs 
the  dies  to  the  nominal  dimensions,  stamps  out  the  parts,  and  rivets  them  together.  They 
identify  flaws  and  determine  which  dies  to  modify  that  will  fix  the  issue,  are  the  least 
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expensive  to  modify,  and  will  yield  the  best  fit.  The  final  dies  and  parts  are  the  masters 
and  the  new  reference  for  the  design.  From  these  masters,  they  update  the  CAD 
specifications  to  match  the  final  part  dimensions,  instead  of  iterating  the  dies  to  meet  the 
CAD. 

The  third  element  is  Seek  Conceptual  Robustness,  as  stated  by  Sobek  et  al. 
(1999).  A  conceptually  robust  design  will  perform  as  intended  in  the  face  of  final  design 
uncertainty;  therefore,  it  has  a  level  independence  from  the  final  selected  solution  by 
other  subsystems  or  functions  (Sobek  et  al.  1999).  When  achieved,  design  works 
regardless  of  what  the  rest  of  the  team  decides  to  do  and  can  be  a  great  enabler  to 
concurrent  engineering  efforts  (Sobek  et  al.  1999).  It  also  includes  seeking  a  design  that 
is  tolerant  to  variations  in  market  conditions  (Sobek  et  al.  1999).  Other  benefits  of 
conceptual  robustness,  noted  by  Sobek  et  al.,  are  that  it  can  collapse  development  time, 
improves  serviceability,  and  is  more  easily  upgradeable  (1999). 

One  example  of  Conceptual  Robustness  in  design  is  the  strategy  undertaken  by  a 
Toyota  radiator  supplier  (Sobek  et  al.  1999).  The  supplier  started  by  optimizing  the 
cooling  core  design,  then  separating  the  radiator  into  the  cooling  core,  upper  and  lower 
tank  sections,  and  ancillary  hoses  (Sobek  et  al.  1999).  They  then  redesigned  the  entire 
production  line,  in  order  to  produce  and  finish  any  size  or  customized  combinations  of 
these  major  subcomponents  on  the  line  (Sobek  et  al.  1999).  The  result  was  a  production 
line  that  tolerated  variations  in  market  conditions  (the  needs  of  their  clients)  and 
performed  well  regardless  of  the  final  design  chosen  (Sobek  et  al.  1999). 

3.  Establish  Feasibility  before  Commitment 

The  third  principle  is  to  Establish  Feasibility  before  Commitment  (Sobek  et  al. 
1999).  The  first  two  principles,  and  according  to  Sobek  et  al.,  Toyota’s  SBD  approach, 
support  the  third  principle  of  establishing  feasibility  before  committing  (Sobek  et  al. 
1999).  The  three  elements  of  this  principle  deal  with  Narrowing  Sets  Gradually  while 
Increasing  Detail,  Staying  within  Sets  Once  Committed,  and  Controlling  by  Managing 
Uncertainty  at  Process  Gates  (Sobek  et  al.  1999). 
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Narrowing  Sets  Gradually  while  Increasing  Detail,  is  fundamental  to  reducing  the 
significant  number  of  sets.  Unlike  PBD,  which  focuses  on  ranking  alternatives  and 
selecting  one  alternative  for  further  development,  SBD  looks  to  reject  the  least  desirable 
solutions  over  time,  thereby  reducing  the  risk  of  incorrectly  eliminating  potentially 
suitable  solutions  until  obtaining  greater  confidence  (Raudberget  2010).  As  the  number  of 
sets  under  consideration  narrows,  development  teams  progressively  elaborate  each 
remaining  set  to  enable  better  understanding  of  the  relevant  differences  before 
committing  (Sobek  et  al.  1999).  According  to  Raudberget  (2010),  scoring  and  screening 
are  methods  of  narrowing  the  number  of  sets;  also,  adding  more  constraints  to  help 
eliminate  sets  when  multiple  solutions  meet  current  requirements. 

Staying  within  Sets  Once  Committed,  bounds  the  development  effort  and  is 
critical  to  SBD,  according  to  Sobek  et  al.  (1999).  They  continue,  stating  that  since 
designers  are  working  concurrently  and  not  fixing  specifications,  except  the  upper  and 
lower  bounds,  it  is  critical  that  design  teams  stay  within  established  sets  of  alternatives. 
Further,  one  development  effort  must  have  confidence  that  another  effort  will  not  jump  to 
a  solution  outside  the  communicated  sets. 

The  third  element,  according  to  Sobek  and  his  colleagues  is  Control  Uncertainty 
by  Managing  at  Process  Gates,  intends  to  reduce  uncertainty  as  needed,  when  needed 
(1999).  They  point  out  that  at  Toyota,  uncertainty  includes  the  size  and/or  number  of  sets 
under  consideration  and  the  level  of  detail  attained.  They  observed  control  for  the  number 
of  sets  and  level  of  specificity  at  design  process  gates.  Further,  they  described  that  the 
required  knowledge  obtained  and  the  number  of  sets  vary  according  to  the  nature  of  the 
subsystem  or  component  being  developed.  For  example,  at  Toyota,  they  observed  that 
due  to  the  complex  nature  and  long  lead  of  the  transmission  subassembly,  the 
“transmission  gate”  is  much  earlier  than  other  subassemblies.  As  a  result,  Sobek  and  his 
associates  stated  that  Toyota  selects  the  transmission  design  years  in  advance  of  the  start 
of  production  (1999).  In  contrast,  they  indicated  that  the  exhaust  system  remains  largely 
undetermined  when  the  transmission  is  fixed.  They  found  that  Toyota  would  allow  the 
exhaust  system  design  to  slowly  narrow,  finalizing  the  design  only  months  before 
beginning  production  (Sobek  et  al.  1999). 
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C.  SEVEN  CHARACTERISTICS  OF  SET-BASED  THINKING 

Using  the  principles  learned  from  Toyota’s  product  development  and  various 
other  applications  of  SBD,  Ghosh  and  Seering  developed  seven  characteristics  of  set- 
based  product  development  in  their  more  contemporary  exploration  of  SBD.  They  took 
the  characteristics  and  further  boiled  them  down  to  what  they  considered  the  two  major 
principles  of  set-based  thinking:  considering  sets  of  alternatives  concurrently  and 
delaying  convergent  decision  making  (2014).  These  principles  are  no  surprise,  based  on 
previous  studies  of  Toyota,  but  the  characteristics  of  set-based  thinking  are  helpful  to 
understand  the  application  of  SBD  better  and  for  making  recommendations  for  the 
implementation  of  SBD  into  the  DOD  acquisition  life  cycle. 

1.  Emphasis  on  Frequent  Low-Fidelity  Prototyping 

Frequent,  low-fidelity  prototyping  is  the  idea  that  producing  several  design 
prototypes,  without  much  detail,  as  Toyota  has  done,  significantly  improves  the  overall 
system  design.  This  sentiment  is  echoed  by  Ward  et  al.’s  explanation  of  Toyota’s 
excessive  number  of  prototypes  and  their  contribution  to  creating  more  robust  designs 
“faster  and  cheaper”  (1995,  44).  Admiral  Richardson’s  sentiment  of  failing  early  and 
failing  often  embodies  this  notion  of  frequent  low-fidelity  prototyping  that  Ghosh  and 
Seering  (2014)  deem  so  important  to  set-based  thinking.  The  number  and  variety  of 
prototypes  open  the  design  space  vice  limiting  it,  allowing  for  the  design  discovery  and 
intersections  of  solutions  to  come  together,  as  Sobek  et  al.  (1999)  described  as  their 
second  principle.  They  are  supported  by  Ghosh  and  Seering  when  they  stated,  “Notably, 
the  proliferation  of  prototyping  throughout  the  design  process  is  a  clear  manifestation  of 
concurrent  development,  as  multiple  prototypes  help  designers  explore  multiple  concepts 
-  which  Toyota  clearly  understood  and  practiced”  (2014,  3).  These  concepts  comprise  the 
whole  set  of  possibilities  for  consideration  and  grants  the  design  team  creative  license. 

Another  significant  aspect  of  prototyping,  as  described  by  Ghosh  and  Seering 
(2014),  is  the  emphasis  of  a  low-fidelity  prototyping  process,  reducing  the  cost  of  each 
prototype,  while  still  allowing  progress  to  continue  in  the  product  development  process. 
These  rapid,  low-fidelity  prototypes  also  avoid  design  fixation.  Design  fixation  is  akin  to 
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the  phenomenon  known  as  “tunnel  vision.”  When  the  designers  fixate  on  a  particular 
design,  which  may  not  be  the  best  solution,  they  lose  sight  of  other,  better  quality 
designs:  “Furthermore,  recent  studies... demonstrate  that  design  fixation  can  be  mitigated 
by  generating  rapid  prototypes.  Thus,  by  mitigating  design  fixation,  designers  enable 
themselves  to  consider  a  wider  range  of  available  options”  (Ghosh  and  Seering  2014,  3). 
The  idea  of  including  a  “wider  range”  of  options,  while  eliminating  infeasible  designs,  is 
a  major  tenet  of  SBD,  as  also  described  by  Sobek  and  his  colleagues  (1999). 

Sets  of  designs,  as  represented  through  prototypes,  help  to  ensure,  not  only  an 
overall  cost  savings  for  the  project,  in  part  due  to  the  low-fidelity,  but  a  better  solution 
and  a  more  robust  design  because  of  the  accelerated  learning  from  rapid,  early 
prototyping.  Admiral  Richardson  clearly  sees  the  value,  hence  the  need  to  work  these 
processes  into  DOD  acquisition. 

2.  Tolerance  for  Under  Defined  System  Specifications 

Tolerance  for  Under  Defined  System  Specifications  is  having  comfort  with  the 
lack  of  detail  communicated  prior  to  design,  similar  to  the  ambiguous  communications  as 
mentioned  by  Ward  et  al.  and  their  description  of  the  “Toyota  Model”  (1995).  Contrary  to 
the  more  traditional  method  of  PBD,  Ghosh  and  Seering  explain  the  value  of  Toyota 
delaying  design  decisions  and  “not  lock[ing]  down  specifications  as  soon  as  possible,”  as 
other  Japanese  and  U.S.  automakers  have  done  (2014,  3).  Under  defined  specifications 
allow  for  flexibility  and  overall  cost  savings,  as  the  design  can  progress,  rather  than  going 
back  to  the  drawing  board,  a  sentiment  echoed  by  Singer  et  al.  (2009).  This  approach  also 
allows  the  project  to  stay  on  schedule  because  they  can  continue  to  make  progress  as 
there  is  no  need  to  “start  over”  while  they  increase  the  level  of  detail  for  the  system 
specification.  Design  flexibility  was  present  in  the  development  of  an  airport  Ghosh  and 
Seering  studied.  The  major  lesson  learned  from  the  airport  case  study  was  that  the 
flexibility  that  was  afforded,  due  to  delaying  decisions,  fostered  an  environment  for 
concurrent  design  sets,  which  ultimately  “mitigate[d]  their  exposure  to  risk  from  events 
such  as  shifting  requirements  or  availability  of  materials”  (Ghosh  and  Seering  2014,  3). 
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By  refusing  to  commit  to  design  specifications  early  and  delaying  decision  in  the  product 
development  process,  flexibility  emerges,  the  design  space  broadens,  and  risk  lowers. 

3.  More  Efficient  Communication  among  Subsystems 

Both  reduced  time  and  cost  are  the  advantages  of  more  efficient  communication 
among  entities  working  various  subsystems.  The  difference  between  point  based 
communication  and  set  based  communication  is  the  increased  time  it  takes  to  interact 
with  all  stakeholders,  as  well  as  the  number  of  required  iterations  to  successfully 
communicate  and  settle  on  a  solution.  “Ward  et  al.  found  support  for  [these]  arguments  in 
Toyota’s  product  development  processes,  where  Toyota  and  its  suppliers  were  found  to 
establish  communication  with  each  other  less  frequently  for  a  shorter  total  duration  of 
time  than  their  U.S.  counterparts  employing  traditional  design  methods,  constant 
communication  among  collocated  engineering  teams  was  a  given”  (Ghosh  and  Seering 
2014,  4).  Essentially,  bi-product  of  effective  SBD  employment  is  more  efficient 
communication  between  engineering  teams  working  various  subsystems.  Furthermore, 
speaking  of  the  construction  field,  they  state  “that  the  mling  paradigm  in  the  construction 
industry  is  a  traditional,  PBD  approach  featuring  long  delays  in  passing  designs  to 
different  agents  in  the  design  process”  (2014,  4).  More  efficient  communications  pave  the 
way  for  a  more  succinct,  rapid  development  of  a  system. 

4.  Emphasis  on  Documenting  Lessons  Learned/Knowledge 

Set  based  thinking  depends  heavily  upon  documenting  lessons  learned  and 
building  a  vast  knowledge  base  to  apply  to  future  design  development.  The  accrued 
technical  knowledge,  as  Ghosh  and  Seering  (2014)  call  it,  allows  mapping  of  the  design 
space,  a  principle  formed  by  Sobek  et  al.  (1999).  Additionally,  the  “lessons  learned 
books”  from  previous  years  of  Toyota  body  designs,  “which,  developed  over  fifteen  years 
at  that  point,  provide  detailed  knowledge  of  what  potential  designs  can  (and  cannot)  be 
implemented”  (Ghosh  and  Seering  2014,  5).  Important  emphasis  is  on  the  potential 
designs  that  “cannot  be  implemented.”  This  sentiment  shows  how  documenting  lessons 
learned  specifically  relates  to  Ward’s  idea  of  narrowing  sets  (1995),  as  well  as  the 
previously  quoted  Sobek  et  al.  study,  which  states  that  SBD  is  about  “determining  what 
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cannot  be  done  or  should  not  be  done”  (1999,  73).  If  lessons  learned  are  not  documented 
and  not  influencing  designs,  the  lessons  will  have  resurface,  which  will  have  a  negative 
impact  on  cost  and  schedule.  That  said,  documenting  lessons  learned  should  improve  life 
cycle  costs  and  timeline.  Keeping  lessons  learned  and  conserving  the  corporate 
knowledge  allows  for  a  more  successful  implementation  of  SBD. 

5.  Support  for  Decentralized  Leadership  Structure  and  Distributed,  Non- 
collocated  Teams 

The  way  Toyota  does  business  with  its  suppliers  is  a  perfect  model  of  a 
decentralized  leadership  structure  in  that  the  suppliers  are  not  provided  with  requirements 
or  specifications,  they  are  allowed  to  make  decisions  based  on  what  they  perceive  the 
needs  of  Toyota  to  be  (Ghosh  and  Seering  2014).  Suppliers’  autonomy  to  work 
independently,  and  in  a  non-collocated  fashion,  is  one  of  the  advantages  of  SBD.  Ghosh 
and  Seering’s  proof  of  the  success  of  a  decentralized  leadership  structure  in  the  software 
development  industry  provides  another  example.  The  “Scrum”  methodology  requires  the 
division  of  labor  into  “Scrum  Teams.”  Though  normally  collocated,  they  describe  a  case 
study  “tracking  a  distributed  team  of  56  developers  across  three  countries  and  witnessed] 
the  most  productive  Java  development  project  to  have  been  documented  [up  to  that  point] 
-  a  testament  to  set-based  product  development  practices  supporting  distributed,  non- 
collocated  teams”  (Ghosh  and  Seering  2014,  6).  Decentralized  leadership  and  distributed, 
non-collocated  teams  goes  hand  in  hand  with  set-based  thinking,  as  it  fosters  the  first 
principle  of  SBD:  “mapping  the  design  space.”  It  provides  autonomy,  which  in  turn 
promotes  creativity  to  define  the  feasible  region,  explore  tradeoffs  with  multiple 
alternatives,  and  communicate  the  set  of  possibilities. 

6.  Supplier/Subsystem  Exploration  of  Optimality 

By  partitioning  teams  and  decentralizing  the  leadership  structure,  the  activities 
developing  various  subsystems  begin  to  take  complete  ownership  of  their  piece  of  the 
system.  When  individuals  take  ownership  of  a  subsystem,  they  are  committed  to  making 
it  the  most  optimal  solution  as  they  can.  As  Ghosh  and  Seering  explain,  “[it]  provides 
subsystems  with  greater  autonomy  in  the  design  process,  [encouraging]  suppliers  and 
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subsystems  to  take  initiative  in  exploring  optimality”  (2014,  6).  When  teams  have  the 
opportunity  to  explore  optimality,  greater  growth  in  technology  and  “breakthroughs”  in 
product  development  occur  (2014).  When  each  team  or  subsystem  works  toward  a  more 
optimal  solution,  the  overall  design  becomes  more  optimal. 

7.  Supports  Flow-Up  Knowledge  Creation 

Because  of  the  decentralized  leadership  structure  and  distributed,  non-collocated 
teams,  communication  flow  reverses  direction,  from  top-down  to  bottom-up,  making  the 
principles  of  SBD  more  applicable.  As  these  teams  are  developing  various  sets  of 
subsystems  and  making  “breakthroughs”  in  technology,  they  communicate  these 
advances  to  the  “top,”  in  this  case,  Toyota.  The  communication  provided  to  Toyota 
allows  them  to  “develop  its  specifications  almost  two  years  [later]”  rather  than  providing 
the  supplier  with  hard  specifications  (Ghosh  and  Seering  2014,  7).  This  broadens  the 
design  space  for  the  suppliers,  providing  a  larger  set  of  possibilities.  This  style  of 
organization,  which  allows  for  “flow-up  knowledge  creation,”  cultivates  the  principles  of 
SBD  and  will  most  certainly  aid  in  their  implementation  into  the  acquisition  community. 

D.  POTENTIAL  COST  AND  SCHEDULE  BENEFITS  OF  SBD 

Though  it  is  difficult  to  distinctly  state  that  the  employment  of  SBD  results  in 
cheaper  systems  acquisitions  for  the  DOD,  there  are  several  ways  in  which  SBD  has  the 
potential  to  save  money.  One  of  the  more  significant  ways  SBD  can  prove  to  be  more 
affordable  is  the  reduction  of  rework.  Kennedy  et  al.  present  the  idea  of  reducing  rework 
through  the  use  of  SBD  (2013).  They  explain,  “rework  that  occurs  late  in  the  product  life 
cycle  is  dramatically  more  expensive  than  design  work  performed  early  in  the  cycle” 
(2013,  1).  Utilizing  SBD  principles  presented  in  this  chapter  will  help  to  eliminate  the 
drivers  of  rework.  By  studying  dozens  of  companies,  they  learned  that  the  primary  causes 
of  rework  can  be  classified  into  three  general  categories: 

[1]  The  development  team  leams  something  critical  late  in  the 
development  process  that  invalidates  prior  assumptions  or  otherwise 
causes  the  team  to  revisit  a  prior  decision  [2].  The  development  team 
makes  critical  decisions  too  early  in  the  project,  before  they  have  the 
knowledge  needed  to  make  a  reliable  decision  [3].  Development  team 
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members  with  one  expertise  inadvertently  make  decisions  that  overly 
constrain  those  of  another  expertise.  (Kennedy  et  al.  2013,  4) 

These  three  categories  relate  directly  to  the  SBD  principles  and  characteristics 
already  covered.  The  first  two  categories  go  hand  in  hand  with  Toyota’s  practice  of 
delaying  critical  decisions  until  more  is  learned,  and  therefore  enabling  a  more  robust 
decision.  While  more  efficient,  set  based  communications,  described  by  Ghosh  and 
Seering,  would  help  alleviate  the  third  of  these  categories.  Going  along  with  the  three 
causes  of  rework,  Kennedy  et  al.  present  three  remedies  for  rework.  They  include 
accelerated  learning,  delaying  critical  decisions  until  sufficient  knowledge  is  learned,  and 
the  application  of  set -based  concurrent  engineering  (2013). 

Kennedy  and  his  coauthors  herald  the  Wright  brothers’  development  of  the  first 
airplane  as  one  of  the  better-documented  examples  of  using  set-based  practices  to  prevent 
rework.  Their  early  work  in  the  development  of  the  airplane  was  more  successful, 
quicker,  and  cost  less  than  the  work  of  other  less  successful  developers  like  Otto 
Lilienthal,  Clement  Ader,  Hiram  Maxim  and  others  (2013).  Table  2,  extracted  from 
Kennedy  et  al.,  shows  the  contrast  in  timeline  and  cost  between  the  early  airplane 
development  activities,  for  which  none,  other  than  the  Wright  brothers,  successfully 
achieved  powered  manned  flight. 


Table  2.  Powered  Manned  Flight  Development  Cost  and  Time 
Comparison.  Adapted  from  Kennedy  et  al.  (2013,  6-7). 


Timeline  (years) 

Cost  ($) 

Wright  Brothers 

4  (22  months  of  actual  work) 

<$1,000 

Otto  Lilienthal 

11 

No  data 

Clement  Ader 

25 

$120,000 

Hiram  Maxim 

~10 

$200,000 

Samuel  Langley 

16 

$70,000 

The  Wright  brothers  achieved  both  the  shortest  timeline  and  cheapest  overall  cost. 
Kennedy  et  al.  attributes  this  to  their  “accelerated  learning”  and  set-based  practices 
(2013).  They  changed  the  approach  from  a  point-based  method  to  a  more  set-based 
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method,  which  included  more  testing  of  ideas  and  prototypes  up  front.  The  point-based 
method  described  by  Kennedy  and  coauthors  was  the  “traditional  design-build-test  cycle” 
(2013,  6).  Instead  of  taking  this  approach,  the  Wright  brothers  aimed  to  leam  more  about 
aerodynamics,  so  they  designed  and  built  the  first  wind  tunnel  to  test  various  wing 
designs,  so  that  they  could  learn  critical  information  upfront  before  building  full-scale 
airplanes.  “Their  focus  was  on  learning  first  via  careful  testing  of  a  variety  of  alternative 
wing  designs”  (Kennedy  et  al.  2013,  6).  One  of  the  seven  characteristics  of  set-based 
thinking  includes  low  fidelity  prototypes,  which  is  exactly  what  the  Wright  brothers  did 
with  their  wind  tunnel. 

With  these  observations,  Kennedy  and  his  contemporaries  describe  practices  to 
reduce  the  likelihood  and  amount  of  rework.  They  recommend  three  set-based  practices 
to  reduce  rework: 

[1]  Replace  the  design-test-build  cycle  with  the  test-before -design  to 
accelerate  learning  in  the  early  phases[2].  Specify  customer  and  business 
interests  as  target  ranges,  giving  the  development  teams  room  to  explore, 
innovate,  and  find  the  most  appealing  tradeoffs[3].  Leverage  set  based 
knowledge  to  communicate  the  key  issues  from  one  area  of  expertise  to 
another.  (Kennedy  et  al.  2013,  16) 

These  recommendations  have  the  potential  to  reduce  rework  and  in  turn  reduce 
the  overall  cost  and  timeline,  as  seen  with  the  Wright  brothers’  success. 

E.  REVIEW  OF  SET  BASED  DESIGN  CONCLUSIONS 

The  SBD  methodology  has  been  juxtaposed  to  the  traditional  PBD  strategy, 
analyzed  for  major  principles,  and  broken  down  into  characteristics.  The  principles  Sobek 
et  al.  have  provided  are  at  the  heart  of  SBD.  They  explain  the  importance  of  considering 
the  set  of  all  possible  solutions,  narrowing  the  possibilities  by  defining  intersections  of 
the  feasible,  and  reducing  the  number  of  solutions  only  as  they  become  infeasible,  or  not 
desired  for  some  reason.  Ghosh  and  Seering’s  characteristics  serve  as  a  “how  to”  guide 
for  implementing  SBD  in  any  organization.  The  first  four  characteristics  are  helpful  when 
defining  processes  for  SBD’s  three  major  principles  found  in  the  Toyota  studies.  The  last 
three  characteristics  focus  on  an  organizational  model  to  facilitate  the  various  processes. 


26 


Ghosh  and  Seering  provide  a  helpful,  contemporary  view  of  what  SBD  is  and 
complements  the  principles  of  SBD  that  have  been  developed  over  the  past  couple 
decades  by  Ward,  Liker,  Sobek,  Doerry,  and  many  others.  The  foundation  they  have  all 
set  with  respect  to  the  understanding  of  what  SBD  is  will  be  invaluable  in  showing  how 
to  implement  SBD  into  DOD  acquisitions.  Ghosh  and  Seering  leave  the  reader  with 
several  questions  relating  to  the  future  works  in  SBD  and  determining  when  SBD  is  best 
suited.  Through  the  examination  of  several  DOD  case  studies  in  the  next  chapter,  the  goal 
is  to  try  to  answer  some  of  their  questions  as  it  applies  to  DOD  acquisitions  and  to 
provide  some  recommendations  on  how  to  change  existing  regulations  to  make  it  work. 
Kennedy  et  al.  also  provide  a  concrete  example  of  how  SBD  has  the  potential  to  increase 
cost  savings  and  decrease  project  timeline. 

Figure  6  presents  the  three  principles  and  seven  characteristics  visually,  while 
showing  how  each  of  them  either  enables  each  other  or  depends  on  one  another  to  work. 
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Principles 


Characteristics 


Frequent  Low-fideity  Prototyping 

Under-defined  Specifications 


'Supporting 

Activities'* 


Eff  i  ci  ent  Co  m  m  un  icat  ions 


Documenting  Lessons  Learned 

Decentralized  Leadership  Structure 

Exploration  of  Optimality 

Flow-up  Knowledge  Creation 


Figure  6.  Principles  and  Characteristics  of  SBD. 

The  “enablers,”  as  listed  in  the  characteristics,  are  what  allow  the  use  of  the 
principles  of  SBD,  while  the  “supporting  activities”  show  which  of  the  characteristics 
support  the  others.  These  connections  show  the  similarities  and  dependencies  they  share 
and  to  help  visualize  these  during  the  case  study  exploration  in  the  next  chapter. 
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III.  SET  BASED  DESIGN  CASE  STUDIES 


In  order  to  understand  fully  how  SBD  has  fit  into  naval  acquisitions  in  the  past,  it 
is  useful  to  review  case  studies  in  which  the  Navy  has  made  an  effort  to  employ  SBD 
principles  and  practices.  From  these  case  studies,  one  can  glean  lessons  to  determine 
which  aspects  or  methods  of  SBD  implementation  have  been  successful  and  those  that 
did  not  work  so  well.  The  following  case  studies  include  the  Ship  to  Shore  Connector 
(SSC),  the  Amphibious  Combat  Vehicle  (ACV),  the  Small  Surface  Combatant,  and  the 
Large  Displacement  Unmanned  Underwater  Vehicle  (LDUUV).  With  the  knowledge  of 
these  projects,  the  goal  is  to  take  away  lessons  to  better  shape  acquisitions,  by 
determining  how  to  implement  SBD. 

A.  SHIP  TO  SHORE  CONNECTOR 

According  to  Buckley  et  al.  (2011),  the  first  application  of  SBD  in  the  Navy  was 
the  SSC  project.  He  explains  that  when  Vice  Admiral  Paul  Sullivan  met  with  Deputy 
Assistant  Secretary  of  the  Navy  for  Ship  Programs  and  Program  Executive  Officer  for 
Ship  Programs,  they  decided  to  begin  government-led  PD  and  CD.  However,  they  desired 
to  complete  the  schedule  in  under  three  years,  therefore  choosing  to  use  SBD  to  “speed 
the  process  for  analyzing  craft  and  systems  alternatives  early  in  the  design  and  also  allow 
consideration  of  more  of  these  alternatives”  (Buckley  et  al.  2011,  79).  They  intended  to 
speed  up  the  process  while  maintaining  design  flexibility  through  understanding  the 
design  space,  integrating  by  intersection,  and  establishing  feasibility  before  commitment. 

The  SSC  is  the  next  generation  Air  Cushion  Vehicle  expected  to  replace  the 
Landing  Craft,  Air  Cushion  (LCAC).  They  reported  that  at  the  completion  of  LCAC 
prototype,  91  craft  were  delivered  to  the  Navy  from  1984  through  2000,  and  they  noted 
that  the  current  LCACs  began  phasing  out  of  service  in  2015.  SSC’s  requirements  are 
similar  to  the  LCAC  to  “transport  equipment,  personnel,  and  cargo  from  ships  located 
over  the  horizon,  through  the  surf  zone,  to  landing  points  beyond  the  high  water  mark  in  a 
variety  of  environmental  conditions”  (Buckley  et  al.  2011,  80).  They  pointed  out  that  if 
weight  is  overloaded,  the  craft  sacrifices  fuel  and  thus  speed.  Table  3  is  a  comparison  of 
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specification  requirements  between  the  LCAC  and  SSC,  showing  service  life,  load 
capability,  speed,  sea  state,  and  distance  from  shore. 


Table  3.  LCAC  and  SSC  Capability  Comparisons.  Adapted  from 

Buckley  et  al.  (2011). 


LCAC 

SSC 

Service  Life 

20- year 

30-year 

Load  Capability 

60  tons 

74  tons 

Speed 

35  knots 

35  knots 

Sea  State 

3 

3 

Distance  from  Shore 

15  nautical  miles 

25  nautical  miles 

Buckley  et  al.  (2011)  explained  that  the  government  led  team  began  PD  in  April 
2008  in  hopes  of  completing  it  in  12  months.  They  added  that  the  short  timeframe  led  the 
team  to  attempt  the  SBD  process.  They  also  indicated  that  the  previous  AoA  completed 
in  November  of  2007  was  a  successful  Gate-2  review  of  the  2-Pass/6-Gate  process.  (For 
readers  unfamiliar  with  the  Navy’ s  Systems  Engineering  and  Technical  Review  Process, 
a  brief  explanation  is  provided  in  Chapter  IV.)  The  SSC  Design  Integration  Team  (DIT) 
consisted  of  the  Ship  Design  Manager,  the  Deputy  Ship  Design  Manager,  and  the  Design 
Integration  Manager.  After  DIT’s  approval  of  the  internal  requirements,  they  entered 
them  into  the  Dynamic  Object  Oriented  Requirements  System  (DOORS),  a  commercially 
available  requirements  traceability  application.  They  noted  that  the  design  team  used  the 
ICD,  the  AoA  Final  Report,  and  the  Resources,  Requirements  Review  Board  (R3B)  to 
bind  the  requirements.  They  developed  the  Capabilities  Development  Document  (CDD) 
at  the  same  time. 

After  developing  the  CDD,  Buckley  et  al.  (2011)  write  that  the  team  would  use 
the  ICD,  AoA,  R3B,  LCAC  specifications,  and  lessons  learned  to  create  the  Functional 
Design  Document  (FDD).  This  FDD  “was  the  set  of  operational  requirements  and 
derived  parameters  used  to  initiate  the  design  effort”  (Buckley  et  al.  2011,  83).  They 
added  the  FDD  is  used  to  create  the  Functional  Requirements  Document  (FRD),  which 
captures  evolving  assumptions  and  requirements  for  an  element  in  a  trade  space  and 

contains  the  element- specific  requirements.  They  indicated  the  FDD  and  FRD  were  used 
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to  plan  for  PD  as  well  as  to  create  the  draft  SSC  specification  after  being  mapped  to  the 
Ship  Work  Breakdown  Structure  (SWBS).  Once  requirements  were  determined  through 
the  SWBS  the  SBD  section  took  place.  Figure  7  shows  the  preliminary  design  schedule 
with  the  SBD  portion  in  the  plans  prior  to  PD  (Buckley  et  al.  2011). 


Integration  Pr°P°*ed 

Subsystem  Trade  Studies  Period  Baseline  PD-1  po-2  Review  CD  Prep 


Apr  21  Jun  21  Aug  18  Sep  26  Nov  3  Dec  19  Jan  5  Feb  20  Mar26  May  1 


Month  01  234  5678  9  10  11  12 


Point 

Design 
Output _ 

Baselme 

Figure  7.  SSC  Preliminary  Design  Schedule.  Source:  Buckley  et  al.  (2011,  84). 

During  the  SBD  process,  Buckley  et  al.  (2011)  reported  that  the  designers 
conducted  trade  studies  and  spent  the  six  weeks  following  integrating  the  systems  into  the 
baseline.  The  team  briefed  senior  leadership  the  proposed  baseline,  which  they  concurred 
on  and  carried  forward  into  PD.  During  the  SBD  process,  the  team  communicated  many 
variations  of  solutions  consisting  of  different  levels  of  performance  and  requirements  for 
different  regions  of  interest.  Regions  of  interest  included  speed,  length,  beam  of  craft, 
load  capacity. 
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Split  teams  covered  different  regions  but  continuously  interfaced  with  each  other 
on  their  findings  and  solutions,  in  order  to  eliminate  infeasible  options.  Their  design 
solutions  would  eventually  converge,  while  conducting  regression  testing  at  the 
functional  level  (Buckley  et  al.  2011). 

Furthermore,  Buckley  et  al.  (2011)  write  that  the  SBD  phase  was  broken  up  into 
three  steps  including  trade  space  setup  and  characterization,  trade  space  reduction,  and 
integration  and  scoring.  They  go  on  to  explain  that  the  trade  space  setup  is  similar  to 
mapping  the  design  space  defined  in  Chapter  II.  They  write  that  the  trade  space  setup  and 
characterization  created  Trade  Space  Summaries  (TSS)  that  characterizes  the  element 
trade  spaces,  operational  requirements,  element  specific  attributes,  and  Technical  Warrant 
Holders  interaction  results.  They  added  the  TSSs  also  track  progress  of  trade  space 
reduction,  and  improve  and  approve  trade  study  plans.  Buckley  et  al.  explain  the  TSS  for 
each  element  captured  all  design  options  with  a  thorough  review.  They  added  that  the 
system  engineer  for  that  element  A  communicates  with  the  system  engineers  from  the 
other  elements  B,  C,  D,  and  all  options  from  A  are  applied  by  the  systems  engineer  from 
B,  C,  D,  etc.,  and  results  of  implementation  are  reported. 

According  to  Buckley  et  al.  (2011),  Element  Trade  Space  Analysis  and  Reduction 
is  the  next  step  in  the  SBD  process  to  determine  acceptable  intersections  of  feasible  sets. 
They  relate  this  step  to  the  Integrating  by  Intersection  principle  described  in  Chapter  n. 
Furthermore,  they  show  use  for  Designs  of  Experiments  (DOEs)  and  other  statistical 
tools  in  this  process.  Furthermore,  Buckley  et  al.  added  Pugh  matrixes  to  explore  trade 
spaces  and  compare  several  designs.  With  these  methods,  the  system  engineers  of  the 
multiple  elements  could  find  the  most  feasible  alternative.  Then  they  used  A  Pareto 
analysis  to  find  the  best  alternative  with  the  lower  cost  and  risk. 

Finally,  engineers  completed  integration  and  scoring  for  the  SBD  process. 

Buckley  et  al.  (2011)  equate  this  step  to  the  SBD  principle  Establishing  Feasibility  before 

Commitment.  They  emphasize  that  at  this  phase  the  trade  space  still  had  100  million 

potential  designs.  They  explain  the  design  team  used  brute  force  method  to  score  and 

integrate  the  final  options  and  that  the  team  then  screened  the  options  for  physical 

viability  to  reduce  the  alternatives  further.  After  completion,  they  compared  the 
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remaining  alternatives  for  value  with  Logical  Decision  (ED)  methodology.  According  to 
Buckley  et  al.,  the  results  of  the  LD,  along  with  subject  matter  experts’  judgment,  led  to 
two  final  designs:  one,  an  aluminum  alloy  craft,  and  the  second,  a  composite  craft  as  SSC 
baselines. 

For  this  case  study,  it  is  well  documented  that  using  SBD  facilitated  a  wide 
variety  of  alternatives  for  review  and  that  this  thoroughly  reviewed  solution  was  chosen 
in  a  much  shorter  than  expected  time  (Buckley  et  al.  201 1). 

B.  AMPHIBIOUS  COMBAT  VEHICLE 

Another  example  of  SBD  in  the  DOD  is  the  U.S.  Marine  Corps  (USMC)  and  the 
ACV  program.  The  ACV  is  a  system  used  to  embark  Marines  from  an  amphibious  ship 
and  land  them  on  the  shore.  Prior  to  the  ACV,  the  USMC  had  been  using  the  Amphibious 
Assault  Vehicle  (AAV)  for  over  40  years.  However,  as  venerable  as  the  AAV  was,  it  was 
aging  and  the  USMC  was  making  way  for  the  more  contemporary  Expeditionary  Fighting 
Vehicle  (EFV),  the  scheduled  replacement  for  the  AAV.  Figure  8  is  the  legacy  AAV  in 
action. 


Figure  8.  Amphibious  Assault  Vehicle.  Source:  Burrow  et  al.  (n.d.,  2). 

Burrow  et  al.  (n.d.)  explain  the  history  of  how  the  ACV  came  about.  They  claim 
that  during  the  development  of  the  EFV,  the  POR  determined  it  to  be  too  costly  and  have 
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excessive  technical  risk,  so  they  cancelled  the  program.  Concurrently,  they  noted  that  the 
USMC  was  studying  the  capability  gaps  of  the  AAV  as  compared  to  the  current  and 
future  concept  of  operations  (CONOPS).  They  were  determined  to  pursue  the  new  ACV. 
However,  in  the  initial  requirements  identification  stage,  they  eliminated  the  need  for  a 
high  water  speed  (HWS)  ACV.  Furthermore,  Burrow  et  al.  state  that  the  lack  of  a  HWS 
capability  was  a  concern  and  made  senior  USMC  officials  reconsider  the  program  and 
embark  on  a  feasibility  study,  based  on  capabilities  trades,  for  a  more  affordable,  HWS 
ACV.  They  wanted  to  determine  if  an  acquisition  program  was  beneficial  for  both  cost 
and  effectiveness  (Burrow  et  al.  n.d.).  It  is  in  this  exploration  of  a  new  ACV  that  we  see 
the  use  of  SBD  principles  and  techniques  that  the  USMC  deployed  for  the  AoA. 

An  ACV  Directorate  was  appointed  to  lead  the  study,  which  commenced  in  2013 
(Burrow  et  al.  n.d.).  They  aimed  to  analyze  four  major  areas:  requirements,  effectiveness, 
trade  space,  and  affordability  (Burrow  et  al.  n.d.).  The  Directorate  partitioned  his 
workforce  into  teams  to  work  in  parallel  using  an  SBD  approach  to  explore  their  areas. 
This  approach  proved  to  be  more  effective  and  less  time  consuming  than  the  point-based 
approach  in  which  each  design  would  be  worked  in  a  linear  fashion,  “tak[ing]  over  a  year 
to  complete,  much  longer  than  the  [six  to  nine]  months  allocated”  (Burrow  et  al.  n.d.,  3). 
According  to  Burrow  et  al.  (n.d.,  2),  the  teams  set  out  to  study  the  following: 

1.  Determine  the  feasibility  and  costs  of  producing  a  HWS  ACV. 

2.  Identify  and  assess  capability  trades  resulting  in  HWS  ACV  procurement 
costs. 

3.  Quantify,  using  modeling  and  simulation,  and  qualify,  using  active  duty 
Marines,  the  operational  benefits  of  a  HWS  ACV. 

4.  Determine  the  differences  in  development,  procurement,  and  operations 
and  support  (O&S)  costs  between  a  low  water  speed  (LWS)  and  a  HWS 
ACV. 

5.  Identify  the  capability  costs  of  a  HWS  ACV,  i.e.,  the  capabilities  that  can 
be  provided  on  a  LWS  ACV  that  cannot  be  provided  on  a  HWS  ACV. 

6.  Evaluate  the  opportunity  costs  of  a  HWS  ACV,  i.e.,  impacts  to  other 
Marine  Corps  programs  and  accounts  required  to  afford  the  HWS  ACV. 
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Another  SBD  principle,  reminiscent  of  Toyota’s  SBCE,  is  that  “definite 
conclusions  would  not  be  made  until  very  late  in  the  study  during  the  comparison  of  the 
alternatives”  (Burrow  et  al.  n.d.,  3).  By  delaying  decisions  until  the  appropriate  time,  they 
analyzed  the  entire  trade  space,  ensuring  all  possible  designs  are  considered.  To  provide 
some  insight  into  how  many  designs  they  studied,  Burrow  et  al.  (n.d.)  stated  that  they  ran 
20  thousand  different  configurations  for  each  of  the  four  capability  concepts,  totaling  80 
thousand  different  designs.  Figure  9  shows  a  depiction  of  a  traditional  approach  to  the 
study  as  compared  to  the  SBD  approach. 


Traditional  Approach 


Determine  Alternative  Affordability 


Compare  Alternatives 


Set-Based  Design  Approach 


Figure  9.  Traditional  Approach  vs.  Set  Based  Design.  Source:  Burrow  et  al.  (2014, 

3). 

The  traditional  method  requires  the  study  of  a  small  number  of  chosen  designs  in 

which  they  employ  the  systems  engineering  process  all  the  way  to  the  AoA.  This  process 

is  iterative  and  time  consuming  by  nature  but  also  restrictive  in  the  design  space.  The 

SBD  model  for  the  study  allowed  the  teams  independently  to  study  the  entire  scope  of 

their  domain,  providing  a  wider  set  of  solutions.  By  working  independently  to  explore  the 

various  sets  of  designs,  the  team  was  able  to  save  time  and  include  a  more  comprehensive 

set  of  possibilities.  The  figure  presents  what  Ghosh  and  Seering  thought  was  so  important 

to  successful  employment  of  SBD:  “Support  for  Decentralized  Feadership  Structure  and 
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Distributed,  Non-collocated  Teams”  (n.d.,  5).  Employing  this  characteristic  effectively  by 
the  ACV  team,  allowed  them  to  finish  the  study  in  under  a  year. 

Since  this  was  a  feasibility  study,  not  all  SBD  principles  or  characteristics  were 
completely  exhausted,  though  they  did  indeed  use  all  three  major.  The  first  principle, 
‘‘Map  the  Design  Space,”  or  “what  cannot  be  done... should  not  be  done,”  eliminated  the 
LWS  region  from  the  feasible  design  region,  as  they  were  specifically  interested  in  an 
ACV  with  HWS  capability  (Burrow  et  al.  n.d.).  Another  example  of  a  Sobek  et  al.  (1999) 
SBD  principle  is  how  the  team  was  able  to  integrate  at  the  intersection  of  these  four  areas 
(requirements  analysis,  effectiveness  analysis,  trade  space  analysis,  and  affordability 
analysis)  to  determine  the  final  outcome  of  the  study.  “The  final  recommendation  for  the 
ACV  would  be  based  on  an  intersection  of  these  four  analyses”  (Burrow  et  al.  n.d.,  3).  By 
dividing  and  conquering,  the  study  was  able  to  accomplish  the  goals  more  efficiently  and 
effectively. 

One  major  assumption  of  the  study  was  that  the  effectiveness  analysis  only 
depended  upon  five  design  components:  water  speed,  personnel  capacity,  weapon  system, 
under-blast  protection,  and  direct  fire  protection  (Burrow  et  al.  n.d.).  They  categorized 
these  design  attributes  as  “big  rocks”  resulting  from  the  fact  they  were  the  largest 
contributors  to  cost  and  weight.  By  doing  this,  the  teams  effectively  mapped  the  design 
space,  by  defining  the  feasible  region.  For  the  trade  space  analysis,  the  assumed  that  only 
HWS  alternatives  were  in  consideration,  and  therefore,  only  varied  the  other  four 
attributes  to  make  thousands  of  different  design  combinations  to  study  (Burrow  et  al. 
n.d.).  Again,  this  is  part  of  the  mapping  principle  of  SBD.  On  the  other  hand,  the 
effectiveness  analysis  team  studied  the  operational  effectiveness  of  each  individual 
attribute  separately,  regardless  of  the  overall  system  configuration  (Burrow  et  al.  n.d.). 
This  exploration  of  the  effectiveness  of  each  system  is  another  example  of  dividing  and 
conquering  through  distributed  teams,  but  also  invokes  the  characteristic  of  exploring  the 
optimality.  The  requirements  analysis  team  studied  all  additional  requirements,  while  the 
affordability  analysis  team  conducted  their  study  as  a  separate  activity. 

The  team  integrated  by  intersection  by  looking  for  the  intersection  of  feasible 

attributes,  through  modeling  and  simulation,  as  well  as  the  principle  of  “Nemawashi.” 
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Much  like  Toyota  using  communication  with  their  suppliers  and  the  “Flow-up 
Knowledge  Creation”  from  Ghosh  and  Seering,  the  team  utilized  surveys  of  250  Marines 
and  held  a  workshop  at  Quantico  for  24  Marines.  They  asked  them  to  rank  and  rate  the 
“big  rocks”  and  other  tradable  capabilities  in  order  of  importance  and  level  of  criticality. 
The  results  allowed  the  team  to  narrow  the  feasible  design  space  by  applying  a  risk  study 
based  on  user  input. 

The  teams  also  invoked  their  own  principle  of  “flexibility,”  similar  to  Ghosh  and 
Seering’ s  “Tolerance  for  Under  Defined  System  Specifications,”  to  minimize  the 
constraints  and  to  control  and  manage  the  uncertainty  at  process  gates.  “Flexibility  is 
defined  for  the  ACV  to  mean  that  for  a  given  requirement,  the  exact  value  for  the 
requirement  has  not  been  established  with  certainty;  the  design  must  be  able  to  affordably 
adapt  to  a  specified  range  for  the  requirement’s  value”  (Burrow  et  al.  n.d.,  14). 

Configuration  modeling  was  the  essential  activity  to  this  study.  In  order  to 
compare  different  design  attributes,  they  defined  “capability  concepts”  as  a  complete 
vehicle  that  possessed  varying  levels  of  “big  rocks,”  along  with  a  list  of  other 
requirements  from  the  requirements  study.  “For  example,  a  capability  concept  would 
refer  to  an  ACV  that  carried  17  troops  and  weapon  system  ‘X’,  and  included  under-blast 
protection  level  ‘C’  and  direct  fire  protection  level  ‘B’”  (Burrow  et  al.  n.d.,  5).  As  stated, 
all  capability  concepts  were  HWS  capable.  These  “big  rocks”  included  various 
combinations  of  troop  capacity,  weapon  system,  and  under-blast  and  direct  fire  protection 
that  were  feasible,  further  narrowing  of  the  design  space.  As  previously  stated,  there  were 
approximately  80  thousand  possible  combinations,  which  is  contrary  to  the  traditional 
approach  in  which  they  only  analyze  up  to  a  few  alternatives  (Burrow  et  al.  n.d.).  They 
did  not  employ  the  traditional  method  in  this  study,  “instead  the  “cloud”  of  all  feasible 
configurations  was  used”  (Burrow  et  al.  n.d.,  5).  The  trade  study  ultimately  yielded  24 
feasible  capability  concepts  as  a  result  of  several  simulations.  These  were  established  by 
exploring  the  “big  rock”  trade-space,  while  the  requirements  study  identified 
approximately  forty  additional,  tradable  requirements,  and  were  analyzed  for  cost  and 
weight  (Burrow  et  al.  n.d.). 
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At  the  conclusion  of  the  study,  in  January  2014,  USMC  officials  were  provided 
with  the  cost,  feasibility,  and  risk  analysis  of  an  ACV  acquisition  program  that  provided 
them  the  ability  to  make  a  well-informed  and  confident  decision  to  pursue  the  ACV.  The 
underlying  theme  for  the  study  was  a  set  based  approach  which  provided  “the  ability  to 
develop  in-depth  knowledge  of  the  technical  problem  and  potential  solution  set,  a  risk- 
based  understanding  of  what  was  feasible  and  infeasible,  and  high  confidence  cost 
estimates  based  on  technical  feasibility  and  diversity  of  solutions”  (Burrow  et  al.  n.d., 
15).  By  employing  SBD,  the  teams  were  able  to  design  a  large  set  of  alternatives,  expand 
the  design  space,  and  provide  a  solid  analysis  for  the  decision  makers,  resulting  in  the 
pursuit  of  the  HWS  ACV. 

C.  SMALL  SURFACE  COMBATANT 

In  2014,  the  Navy  created  a  Small  Surface  Combatant  Task  Force  to  assist  the 
Secretary  of  Defense  in  budget  deliberations.  The  Task  Force  for  the  Small  Surface 
Combatant  had  several  tasks.  First,  the  team  had  to  establish  both  the  requirements  and 
the  trade  space  of  the  Small  Surface  Combatant.  Then  the  team  had  to  consider 
alternatives  for  the  design  concept:  a  modified  Littoral  Combat  Ship  (LCS)  design,  an 
existing  ship  design,  and  a  new  ship  design.  Each  concept  was  to  be  explored  for  four 
major  facets:  top-level  requirements,  cost,  Milestone  schedule,  and  lethality  to  air, 
surface,  and  undersea  threats  (Garner  et  al.  2015,  1).  Figure  10  displays  the  requirements 
analysis  process  that  the  task  force  utilized. 
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Figure  10.  Small  Ship  Combatant  Task  Force  Process.  Source:  Garner  et  al.  (2015, 

3). 


The  task  force  conducted  multiple  parallel  efforts  to  define  the  mission  areas  and 
the  capability  concepts  that  defined  the  requirement  trade  space.  Separate  groups 
characterized  the  threat  environment,  reviewed  the  potential  roles  of  the  SSC,  and 
developed  the  Capability  Concept  Wheel.  All  the  groups  provided  continuous  feedback  in 
order  to  properly  conduct  the  requirements  analysis.  Next,  a  parallel  effort  analyzed  each 
of  the  three  ship  design  efforts:  a  modified  LCS,  a  new  ship  design,  or  an  existing  ship 
design.  They  then  analyzed  all  of  these  alternatives  for  feasibility  along  with  cost  and 
programmatic  concerns. 

Set  Based  Design  was  utilized  throughout  this  process  taking  points  from  the  SBD 
principal  concepts,  as  stated  by  Gamer  et  al.  (2015,  2): 

1.  Consider  a  large  number  of  potential  solutions. 

2.  Have  specialists  evaluate  sets  of  solutions  from  their  own  perspective. 

3.  Intersect  the  sets  to  optimize  a  global  solution  and  establish  feasibility 
before  commitment. 
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These  three  principles  map  closely  to  Toyota’s  original  principles  for  concurrent 
engineering.  The  first  two  SBD  principles  relate  directly  to  Toyota’s  “Map  the  Design 
Space.”  The  task  force  looked  to  consider  a  large  solution  set  while  also  designing 
multiple  alternatives  through  specialists  evaluating  the  solution  set  each  with  their  own 
perspective.  The  last  principle  relates  to  Toyota’s  “Integrate  by  Intersection”  and 
“Establish  Feasibility  before  Commitment.”  The  task  force  found  intersections  of 
feasible  sets  and  stayed  within  the  sets  once  committed. 

The  first  step  the  team  took  was  to  create  a  Capability  Concept  Wheel  as  shown  in 
Figure  11.  Each  wedge  of  the  wheel  has  several  configurations  and  opens  up  the  trade- 
space.  Each  level  in  the  wedge  provided  an  increased  capability.  For  example,  one  of  the 
wedges  on  the  bottom  right  is  Underway  Days.  The  level  closest  to  the  center  is  for  15 
days,  and  it  increases  outwards  to  30,  45,  and  60  days.  This  allowed  the  team  to  evaluate 
the  trade  space  for  the  different  capabilities. 
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Figure  11.  Capability  Concept  “Bullseye”  Chart.  Source:  Garner  et  al.  (2015,  2). 
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The  task  force  focused  on  four  mission  areas:  Air  Warfare  (AW),  Anti-Submarine 
Warfare  (ASW),  Mine  Warfare  (MIW),  and  Surface  Warfare  (SUW).  They  considered 
the  other  elements  enabling  capabilities.  The  team  utilized  the  different  wedges  and 
levels  in  the  wheel  to  create  192  capability  concepts.  Each  of  the  concepts  provided 
different  enabling  capabilities  for  the  Small  Surface  Combatant.  Each  were  then 
examined  and  then  narrowed  down  to  13.  From  there,  they  continued  with  eight  because 
the  ASW  options  were  very  similar.  Table  4  displays  the  eight  capabilities.  Columns  CC1 
to  CC8  list  each  capability  concept.  Each  concept  was  mapped  to  the  mission  area 
capabilities,  and  the  “X”  was  utilized  to  determine  if  the  concept  met  the  mission  area 
capabilities. 


Table  4.  Capability  Concept  Mission  Area  Capabilities.  Source: 

Garner  et  al.  (2015,  4). 


Mission  Area  Capabilities  CC  1 

Self  Defense  against  Air, 

Surface,  Undersea  Threats 

Capability  to  detect  and 
engage  small  craft  within-  thc- 
horizon  of  own  ship 

Capability  to  achieve  mission 
kill  of  over-the-horizon 
surface  targets 

Capability  to  detect  and 

engage  undersea  threats  in  X 

support  of  ASW  operations 

Limited  capability  to  defend 
other  ships  against  ASCMs 


Capability  Concept 

CC  2  CC  3  CC  4  CC  5  CC  6  CC  7  CC  8 


X  X 


X  X  X  X 


X  X 


Combat  system  engineers  then  modeled  all  of  the  concepts,  created  alternatives 

for  each  of  the  capability  concepts,  and  ran  them  through  a  detect-control-engage  kill 

chain  simulation  analysis.  They  estimated  Space,  Weight,  Power,  and  Cooling  (SWAP- 

C),  costs,  and  manpower  to  design  three  options:  a  modified  LCS  design,  an  existing  ship 
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design,  and  a  new  ship  design.  Then  they  analyzed  Feasibility,  cost,  and  for  each  of  the 
designs. 

For  the  LCS  modifications,  several  excursions  were  conducted  which  traded 
enabling  capabilities  (EC)  to  preserve  primary  mission  area  (PMA)  capabilities,  traded 
PMA  performance  to  levels  that  would  still  provide  operational  utility,  and  implemented 
engineering  tradeoffs  among  design  features  to  address  SWAP-C  and  center  of  gravity 
concerns.  This  excursion  analysis  was  an  important  element  in  helping  to  explore  fully 
the  design  trade  space,  as  it  explored  means  to  increase  space,  weight,  power,  or  cooling, 
or  lower  center  of  gravity  to  provide  additional  trade  space  for  capability  concept 
exploration  (Garner  et  al.  2015). 

The  new  ship  design  utilized  Advanced  Surface  Ship  and  Submarine  Evaluation 
Tool  (ASSET)  and  Rapid  Ship  Design  Environment  (RSDE)  to  create  the  design  space  of 
over  15,000  different  configurations.  The  designers  placed  these  configurations  into  the 
Engineering  Resilient  Systems  (ERS)  Trade  space  Toolkit  as  shown  in  Figure  12.  They 
implemented  five  models:  Combat  Systems  calculator,  regression  models,  cost  models, 
feasibility  element  calculator,  and  a  configuration  feasibility  calculator.  Utilizing  a  Monte 
Carlo  simulation,  a  subset  of  values  emerged  that  mapped  to  the  Capability  Concept  and 
the  others  randomly  chosen.  From  those,  the  feasibility  of  each  was  determined  based  on 
the  risk  level  for  operational  success. 
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Figure  12.  ERS  Trade  Space  Toolkit  Structure.  Source:  Gamer  et  al.  (2015,  7). 


By  employing  SBD  processes,  the  task  force  evaluated  thousands  of  design 
alternatives  and  provided  the  leadership  insight  to  make  acquisition  decisions  within  six 
months.  Adhering  to  the  three  methods  allowed  the  task  force  to  analyze  thousands  of 
potential  solutions,  analyze  all  the  design  alternatives  in  parallel,  and  find  intersections  in 
the  feasibility  to  establish  the  overall  operational  risk  of  those  solution  sets.  Specialists, 
such  as  the  combat  systems  engineers,  evaluated  from  their  perspective  and  the  feasibility 
calculator  enabled  the  team  to  establish  feasibility  of  the  solutions  before  commitment  to 
the  design.  The  Secretary  of  Defense  accepted  the  task  force  recommendation  to  build  a 
new  Small  Ship  Combatant  ship  based  on  an  upgrade  to  the  LCS. 

D.  THE  LARGE  DISPLACEMENT  UNMANNED  UNDERWATER 
VEHICLE1 

The  final  case  study  explores  an  SBD  implementation  process,  which  is  currently 
in  progress  for  a  POR.  This  analysis  will  focus  more  on  the  steps  of  implementation  and 
the  types  of  tools  needed  to  employ  SBD  to  a  developing  system.  The  program  of  interest 


1  A  significant  portion  of  the  information  contained  within  this  chapter  is  based  on  personal  knowledge 
of  one  of  the  authors  who  has  worked  within  the  LDUUV  program. 
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is  the  LDUUV  and  the  programmatic  details  shared  will  be  kept  generic  to  avoid 
violating  Defense  distribution  and  dissemination  regulations  of  technical  documentation. 

The  LDUUV  will  provide  the  Fleet  more  undersea  military  capability  in  terms  of 
endurance,  operational  range  and  modular  payloads.  The  LDUUV  will  extend  the 
footprint  of  the  warfighter  with  launch  and  recovery  integration  for  the  Virginia-class  and 
the  Ohio-class  guided  missile  submarines  as  well  as  littoral  combat  ships.  The  program, 
established  by  the  Unmanned  Maritime  Systems  Program  Office,  resides  in  the  Program 
Executive  Office  (PEO)  Littoral  Combat  Ships. 

In  early  2016,  the  PEO  LCS  (Milestone  Decision  Authority)  ceased  the  release  of 
the  Request  for  Proposal  (REP)  to  industry  and  decided  to  have  the  design  led  by  a 
government  team  at  Naval  Undersea  Warfare  Center  (NUWC)  Newport,  in  partnership 
with  other  Warfare  Centers  and  industry  as  necessary  to  design  and  integrate  the  system. 
They  also  wanted  to  use  the  growing  systems  engineering  methodology  of  SBD  to  field  a 
more  reliable  system  through  highly  controlled  design  techniques,  reducing  the 
probability  of  scope  creep,  rework,  and  cost  and  schedule  overruns. 

The  program  was  near  RFP  release,  which  equates  to  being  between  Milestones  A 
and  B;  more  specifically,  post  Gate  4  of  the  Department  of  the  Navy  (DON)  2-Pass/6 
Gate  requirements/acquisition  process.  They  sought  to  implement  SBD  at  this  point.  The 
CDD  was  the  main  tool  to  map  properly  the  design  space,  as  discussed  in  detail  as  one  of 
Toyota’s  three  main  principles.  These  capabilities  form  the  initial  LDUUV  Capability 
Concept  and  facilitate  further  decomposition  and  mapping  efforts  for  Measures  of 
Effectiveness,  Measures  of  Performance,  and  Key  Performance  Parameters.  The  CDD 
was  essentially  the  driver  for  the  LDUUV  design  space  and  furthermore  the  solution  sets. 

A  “Capability  Concept”  as  referenced  above,  combines  a  requirement  set  with  a 
CONOPS.  Changing  the  requirements  and/or  the  CONOPS  will  produce  a  different 
capability  concept  that  will  lead  to  a  different  set  of  solutions. 

The  plan  for  how  to  use  SBD  is  currently  in  flux;  program  schedule  and  funding 
limitations  have  created  an  increasingly  dynamic  environment.  The  high-level  process 
remains  stable  but  the  inputs  and  outputs  of  the  plan  are  not  final.  For  example,  they 
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originally  planned  SBD  at  different  levels  of  system  development.  Level  one  focused  on 
achieving  a  subset  of  the  CDD  based  on  technical  and  stakeholder  prioritization,  as  well 
as  “integrating  by  intersection”  of  four  major  high  impact  tradable  requirements  domains: 
Forward  Looking  Sonar,  Synthetic  Aperture  Sonar,  Energy,  and  Wet  versus  Dry 
Architecture.  Ranges  of  solutions  for  the  four  domains  would  be  paired  down  through 
more  detailed  analysis  (such  as  Model  Based  Systems  Engineering)  to  create  a  final 
design  recommendation  for  prototyping.  The  results  of  the  prototype  testing  influenced 
the  future  SBD  analysis  of  level  two,  which  included  all  the  CDD  requirements,  therefore 
creating  a  larger  trade  space.  Ultimately,  in  level  two,  the  result  of  the  SBD  analysis 
produced  a  second  prototype  that  finalized  the  technical  data  packages  for  low  rate 
production  turnover  to  industry. 

As  the  scope  of  the  SBD  effort  became  more  robust  along  with  the  pressures  of 
the  strict  schedule  and  funding  hardships,  they  realized  that  to  ensure  the  first  prototype  is 
complete  by  the  set  milestone,  SBD  will  be  diverted  from  influencing  the  first  prototype; 
the  first  prototype  will  be  an  accelerated  learning  input  into  the  SBD  process.  This  aligns 
with  Ghosh  and  Seering’s  Tolerance  for  Under  Defined  System  Specifications  (2014). 
This  input  informs  the  design  team  and  decision  makers  of  the  refined  requirements  to 
pursue,  but  the  trade  space  remains  open  until  further  input  and  analysis  is  complete. 

As  stated  earlier,  the  process  for  implementing  SBD  into  the  LDUUV  remains 
somewhat  fixed  and  the  execution  of  this  process  is  still  in  its  infancy,  though  the  finer 
details  for  the  execution  of  SBD  are  not  determined.  Figure  13  is  the  overall  scope  of 
steps  for  using  SBD  as  well  as  showing  the  needed  resources/  tools  to  guide  the  process 
from  inception  through  realization. 
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Figure  13.  LDUUV  SBD  Scope  of  Steps.  Source:  Hardro  (2016,  2). 


SBD  is  an  intricate  process  and  takes  a  great  deal  of  commitment  and  resources  to 
initiate;  this  will  be  apparent  when  working  through  this  scope  of  steps. 

Examining  Figure  13,  one  can  see  that  the  implementation  process  revolves 
around  the  Assessment  Framework  (AF).  The  AF  is  a  framework  used  by  decision 
makers  and  analysts  to  integrate  their  cost  and  technology  models,  execute  them,  and 
visualize  the  results.  This  framework  enables  linkages  between  not  only  evaluation 
models,  but  databases  to  continue  to  pull  the  most  relevant  and  newest  technical  data 
from.  The  AF  is  composed  of  four  distinct  modules:  Ground  Rules  &  Assumptions, 
Technical  Model,  Cost  Model  and  the  Research  Database.  The  Ground  Rules  & 
Assumptions  module  sets  the  rules  for  the  AF.  The  module  defines  the  assumptions, 
reasoning,  and  engineering  judgment  used  to  bind  the  integrated  framework. 

The  Technical  Model  is  the  tool  used  to  evaluate  different  system  or  subsystem 
concept  performance  based  on  the  technical  parameters  of  the  components.  If  considering 
emerging  technologies,  there  needs  to  be  a  methodology  to  capture  risk  associated  with 
certain  technology  readiness  levels.  The  Cost  Model  determines  life  cycle  costs  of 
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systems  or  subsystems  in  terms  of  cost  ranges.  The  Research  Database  looks  similar  in 
format  to  a  Work  Breakdown  Structure  and  breaks  down  systems  and  subsystems  to  a 
low  enough  level  to  list  the  performance  and  technical  characteristics.  Continuously 
populating  this  database  with  new  data,  it  becomes  the  building  blocks  for  the  design 
space  with  the  AF.  This  data  feeds  the  cost  and  technical  models  for  further  analysis. 

The  output  of  the  initial  AF  run  is  a  set  of  potential  configurations,  or  the  first 
round  of  solution  sets,  while  a  more  detailed  analysis  will  start  to  pare  down  the  solution 
sets  further.  Each  one  of  these  modules  requires  careful  integration  to  connect,  formulate, 
and  present  the  data  in  views  so  that  the  decision  makers  and  stakeholders  can  make 
informed  conclusions.  Choosing  the  correct  tools  to  evaluate  the  problem,  integrating  the 
results  from  different  models,  and  displaying  the  solutions  comprehensively  are 
challenging  tasks. 

There  are  other  influences  to  the  considerations  of  the  first  solution  sets  and  those 
are  the  outputs  of  the  Requirements  Database  (RD)  analysis  and  the  Fleet  Valuations 
(FVs).  The  RD  is  the  mirror  image  of  the  CDD,  but  each  of  those  capability  requirements 
are  ranked  via  input  from  the  Stakeholders  into  high  impact,  medium  impact,  and  low 
impact  tradable  requirements  (HITR,  MITR,  and  LITR).  This  ranking  or  “binning”  of 
requirements  puts  more  focus  on  the  HITR  when  utilizing  the  AF.  There  is  an  emphasis 
on  understanding  how  the  requirements  correlate  and  trade,  and  therefore  accounted  for 
in  the  excursions  of  the  AF. 

The  FVs  are  another  major  influence  into  the  AF.  The  valuation  is  a  manner  of 
gathering  Fleet  input  and  priority  into  what  capabilities  are  important  when  running 
LDUUV  missions.  During  the  composition  of  this  report,  the  design  team  held  a  one-day 
workshop  with  given  mission  scenarios  to  gather  Fleet  priority.  Figure  14  depicts  the 
CCW,  a  tool  used  to  gather  the  Fleet  input.  The  CCW  is  very  similar  to  the  Small  Surface 
Combatant  CCW.  Both  focus  on  the  major  capabilities  of  their  respective  systems. 
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Figure  14.  Capability  Concept  Wheel  for  the  LDUUV.  Source:  Hardro  (2016,  5). 


This  CCW  is  still  a  draft  and  will  evolve  as  more  input  is  received  from  the 

NAVSEA  05  (Naval  Systems  Engineering  Directorate)  technical  community.  The  wheel 

represents  four  capability  concepts:  Communicate,  Autonomy  &  Command  and  Control 

(C2),  Mission,  and  Core.  From  there  the  CCW  partitions  down  into  capabilities  and  then 

to  capability  increments.  For  example,  within  the  capability  concept  “Autonomy  &  C2,” 

Adaptive  Decision  Making  and  External  Interaction  are  the  capabilities.  Within  the 

capability  “External  Interaction,”  moving  from  lowest  capable  to  highest  (inside  to 

outside  of  circle),  Remote  Control  Operation,  Semi-Autonomous  Operation,  and  Fully 

Autonomous  Operation  are  the  capability  increments.  The  CCW  then  garners  input  from 

the  Fleet  through  these  capabilities  that  should  encompass  an  entire  LDUUV  unit.  The 
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output  of  the  workshop  is  a  hierarchy  listing  of  capabilities  in  order  of  importance;  this 
information  feeds  the  AF  to  add  another  variable  to  the  decision-makers  view  when 
choosing  the  best  solution. 

After  populating  the  AF  data,  the  designers  generate  the  first  set  of  potential 
configurations  through  the  inputs  and  relationships  built,  tools  chosen,  and  components 
integrated.  According  to  Figure  13,  LDUUV  Scope  of  Steps,  this  is  the  “Baseline 
Analysis”  (Hardro  2016,  2).  After  the  initial  determination,  the  design  space  narrows  by 
adding  more  fidelity  to  the  design  by  means  of  constraints,  more  defined  requirements, 
engineering  judgment,  and  cost  analyses.  They  then  produce  detailed  models  and  analyze 
an  integrated  logistics  support  system.  This  more  detailed  analysis  generates  a  list  of 
feasible  configurations  and  from  there,  low-fidelity  subsystem  prototyping,  tests, 
experimentation,  and  further  detailed  analysis  take  place  to  demonstrate  a  list  of  viable 
configurations.  The  result  is  the  elimination  of  infeasible  sets,  creating  a  global  solution 
and  presenting  the  well-informed  customer  with  more  than  enough  detail  to  make  a 
decision. 

The  LDUUV,  being  in  its  infancy  stages  of  the  SBD  implementation  process  and 
procedures,  continues  to  adapt  to  some  flux,  as  can  be  expected,  as  much  more  detail  and 
fidelity  emerges  over  the  course  of  the  project.  Most  of  the  personnel  working  the  SBD 
methodology  are  new  to  it,  which  presents  a  learning  curve  in  the  process,  leading  to 
even  more  fluctuation.  It  is  apparent  from  the  other  case  studies  that  there  is  no  right  way 
to  implement  SBD.  That  sort  of  advantage  is  beneficial  if  the  team  has  experience  using 
the  process.  To  become  more  proficient  with  this  methodology,  the  acquisition  corps 
needs  to  produce  guidelines  and  assumptions  for  how  to  execute  and  leverage  the 
process.  There  is  also  a  need  for  tools  and  templates  to  build  certain  SBD  products  such 
as  a  resource  database  or  a  capability  concept  wheel  to  provide  repeatability  and  stability, 
as  well  as  support  documenting  and  applying  lessons  learned  across  the  DOD 
community.  The  high  level  SBD  Scope  of  Steps  took  two  months  to  set  up  and  since  its 
inception,  the  tools  utilized  for  the  technical  and  cost  modeling  still  not  finalized.  The 
CCW  is  on  its  third  month  of  development  and  is  still  a  draft.  Changes  to  the  CCW 
continue  as  more  high-level  input  arises.  The  biggest  challenge  so  far  is  not  the 
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engineering  behind  SBD,  but  the  administrative  burdens  in  front  of  it,  though 
stakeholders  approve  and  progress  moves  forward  on  these  products  and  processes.  The 
smaller  issues  are  the  lack  of  experience  utilizing  SBD  and  the  amount  of  legwork  needed 
to  initiate  the  SBD  process  (i.e.,  AF).  As  heard  in  an  undisclosed  meeting,  VADM 
Johnson,  the  principal  Military  Deputy  for  the  Assistant  Secretary  of  the  Navy  for 
Research,  Development,  and  Acquisition,  once  said,  “don’t  cost  us  out  of  the  business.” 
The  administrative  and  oversight  burden  has  potential  to  do  just  that. 

E.  NAVAL  SURFACE  WARFARE  CENTER  CARDEROCK  DIVISION 

POINT  BASED  DESIGN  AND  SET  BASED  DESIGN  COMPARISON 

The  basis  for  the  following  case  study  is  the  slide  presentation  entitled 
“Engineered  Resilient  Systems  for  Ship  Design  and  Acquisition”  by  MacKenna  and  his 
co-authors  (2014).  This  presentation  describes  the  results  of  a  three-month  design 
exercise  to  compare  and  contrast  the  application  of  PBD  and  SBD  methodologies  at 
Naval  Surface  Warfare  Center  Carderock  Division. 

They  divided  the  employees  at  Carderock  into  two  separate  design  teams,  a  PBD 
team  and  a  SBD  team.  Both  teams  were  tasked  to  further  develop  a  provided  baseline 
ship  design  with  each  to  deliver  a  final  ship  design  using  their  respective  design 
techniques.  They  evaluated  the  designs  for  cost,  effectiveness,  and  risk  throughout  the 
exercise.  The  exercise  was  intended  to  compare  the  resulting  designs  and  lessons  learned 
of  using  PBD  versus  SBD  methodologies.  During  the  study,  the  team  introduced  two  sets 
of  requirements  changes  and  a  mid-life  system  upgrade  to  determine  how  resilient  the 
design  process  was  and  how  it  could  adapt  to  the  changes  (MacKenna  et  al.  2014). 

One  significant  finding  from  the  study  as  presented  by  MacKenna  and  his  co¬ 
authors  (2014)  was  SBD’s  effect  on  projected  cost.  At  the  beginning  of  the  SBD 
execution,  the  design  space  is  broad,  and  as  a  result,  there  was  a  wide  range  of  estimated 
costs,  representing  the  various  potential  ship  design  alternatives  (2014).  As  the  design 
space  narrowed,  the  range  of  cost  estimates  narrowed.  Figure  15  depicts  this  narrowing  of 
the  predicted  cost  estimates  as  percentage  of  baseline  ship  design  cost  estimate  versus 
time  for  both  the  PBD  and  SBD  designs. 
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Cost  vs.  Time 


Figure  15.  Cost  as  a  Percentage  of  Baseline  vs.  Time.  Source:  MacKenna  et  al. 

(2014,  14). 

The  black  line  indicates  the  cost  estimate  for  the  evolving  PBD  solution,  while  the 
dashed  red  lines  are  the  upper  and  lower  bounds  of  the  cost  estimate  range  for  the  set  of 
possible  solutions  the  SBD  team  considered.  Initially,  during  the  design  space  mapping, 
SBD  did  not  have  established  cost  estimates  as  they  explored  the  design  space 
(MacKenna  et  al.  2014).  The  Requirements  Change  1  came  about  in  the  middle  of  March 
2013.  At  this  time,  the  two  teams  reduced  the  estimated  cost  for  their  systems  to  just 
above  the  ship  baseline.  Requirements  Change  2  arose  in  early  April;  this  increased  the 
projected  cost  for  the  PBD  to  118  percent  of  baseline  and  the  SBD  range  to  111-122 
percent.  The  change  forced  rework  in  the  point-based  team’s  design,  while  the 
requirements  change  actually  assisted  the  SBD  team  to  eliminate  infeasible  solutions  and 
helped  reduce  the  design  space  (MacKenna  et  al.  2014).  At  the  conclusion  of  the  study, 
the  resultant  SBD  solution  was  4  percent  cheaper  than  the  PBD  solution,  at  1 1 1  percent 
and  115  percent  respectively,  of  baseline  ship  cost. 
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Table  5  summarizes  the  Carderock  design  teams’  experience  applying  the  two 
methodologies  (MacKenna  et  al.  2014,  16). 


Table  5.  PBD  and  SBD  Comparison.  Source:  MacKenna  et  al.  (2014, 

16). 


Point-Based  Design 

Set-Based  Design 

Design  decisions  largely  driven  by  the 
designer’s  preference 

Design  decisions  were  driven  by  design/ 
analysis  data,  with  each  design  decision 
formally  documented 

Design  Decisions  that  were  made  early 
were  largely  set  through  the  process. 

(ship  sizing  and  system  architectures) 

Decision  space  was  open  until  the  end  of 
the  design  process.  Subsystem  design  was 
done  before  the  ship  was  sized,  ship  sizing 
was  one  of  the  last  steps 

Design  progressed  rapidly,  with 
iterations  on  detailed  analysis  happening 
early 

Design  progressed  slowly  at  first,  with 
significantly  more  work  done  up  front,  with 
lower  fidelity  tools,  to  reduce  the  design 
space  to  a  point  where  more  detailed 
analysis  could  be  performed  in  an 
economical  manner 

Requirements  change  caused  significant 
rework 

Requirements  changes  caused  no  rework, 
and  actually  facilitated  the  set  reduction 
process 

As  cost  requirement  decreased  during  the 
experiment,  there  was  not  much 
flexibility  to  adapt.  Without  exploration 
of  the  design  space,  the  point  based  team 
had  to  guess  how  to  achieve  cost 
reduction 

Set  based  process  provide  the  team  with 
robust  information  to  do  MOE  versus 
aggressive  cost  goal  tradeoffs 

Resulting  design:  high  performance, 
complex,  high  risk  design  with  lower 
reliability 

Resulting  design:  high  performance, 
simple,  low  risk,  and  higher  reliability 

The  comparison  displays  some  of  the  key  advantages  of  utilizing  SBD.  The  SBD 
team  kept  the  design  space  open  until  the  end  of  the  design  period.  As  a  result,  the  SBD 
team  adapted  quickly  to  the  requirements  changes  without  the  significant  addition  of 
increased  cost.  Their  design  was  very  analysis  oriented,  using  established  tools  to  analyze 
the  design  space  versus  the  designers’  preference  in  the  PBD  design.  In  the  end,  the  SBD 
was  a  simpler  design  with  lower  cost  and  risk. 
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F.  CASE  STUDY  CONCLUSIONS 

The  case  studies  presented  in  this  chapter  provide  insight  to  how  DOD  utilized 
SBD.  The  most  important  feature  is  the  lack  of  similarity  of  the  implementation  process 
amongst  the  case  studies;  the  absence  of  guidelines  allows  tailoring  SBD  to  the  specific 
needs  of  a  program.  There  are,  however,  generalities  or  common  themes  selected  to 
provide  structure  to  the  process,  therefore  presenting  an  opportunity  to  leverage  these 
factors  when  making  recommendations  for  acquisition  tailoring.  First,  it  is  helpful  to 
summarize  the  key  attributes  of  SBD  from  the  advantages  and  disadvantages  standpoint; 
understanding  these  is  paramount  for  choosing  the  acquisition  strategy.  Table  6  is  a 
breakdown  of  some  of  the  key  SBD  advantages  and  disadvantages. 

Table  6.  Advantages  and  Disadvantages  of  SBD. 

_ Advantages _ 

Establishes  system  design  feasibility  before  commitment 

Design  development  is  not  limited  to  a  single  solution 

Enables  parallel  system  level  development  with  subsystem  level  development 

Delays  decisions  on  system  requirements  until  trade-offs  are  further  understood  while 
continually  promoting  design  discovery 

Flexibility  for  requirements  change 

Concurrent  discipline  design  development  can  take  place  by  remotely  dispersed  design 
team 

Enables  rapid  analysis  of  competing  systems  and  design  alternatives 
Enables  low  risk  and  high  reliability  designs  via  eliminating  infeasible  solutions  first 
Cost  commitments  deferred  until  sufficient  design  detail  permits  selection 
Longer  period  for  stakeholder  influence  and  feedback 

Allows  for  design  flexibility,  reducing  re-work  and  providing  potential  cost  savings 
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Disadvantages 


Higher  initial  costs  upfront  in  the  design  process 


Commitment  of  resources  needed  to  build  Assessment  Framework  and  perform 
integration  of  SBD  tools 


Lack  of  education  and  experience  from  which  to  draw  during  execution  and 
implementation  of  SBD. 


No  guidelines  or  assumptions  for  SBD  process  and  execution 


No  instruction  or  guidelines  for  integration  into  DOD  acquisition 


There  are  multitudes  of  advantages,  hence  the  popularity  and  the  push  to  leverage 
this  methodology.  This  and  the  previous  chapter  discuss  the  first  four  advantages  listed, 
which  are  the  core  benefits  of  SBD.  Set  Based  Design  allows  design  feasibility  before 
commitment,  does  not  limit  design  development  to  a  single  solution,  condenses  overall 
design  development  time  through  multiple  parallel  efforts  and,  most  importantly, 
demonstrates  fidelity  in  system  requirements  by  delaying  decisions  until  more  is 
understood  about  the  trade-offs  (Byers  2016).  These  core  benefits  help  enable  the  Fleet  to 
receive  rapid  reliable  delivery  of  advanced  defense  systems. 

Other  less  touted  advantages  are  in  the  remaining  items  listed.  “Flexibility  for 
requirements  change”  relates  to  the  focus  on  delaying  design  decisions  until  better 
understanding  requirements  (Gray  2015).  The  analysis  and  the  shrinking  of  the  design 
space  accept  requirements  changes  to  help  further  define  the  space.  SBD  allows  for  a 
greater  period  for  evolving  or  changing  requirements  as  compared  to  the  point-based 
method;  PBD  lends  itself  to  costly  scope  creep  due  to  stress  surrounding  the  requirements 
of  the  system.  This  advantage  directly  feeds  “Longer  Period  for  Stakeholder  Influence 
and  Feedback.”  The  postponing  of  design  decisions  allows  the  customer  more  flexibility 
to  adjust  requirements  over  a  longer  period.  With  the  changing  threats,  operational 
environments,  and  advancing  technologies,  having  this  luxury  is  crucial  in  fielding  the 
right  system  at  the  right  time  to  meet  the  evolving  need. 
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“Concurrent  discipline  design  development  can  take  place  by  remotely  dispersed 
design  team”  is  true  and  important  because  subject-matter  experts  and/or  Integrated 
Project  Teams  can  work  remotely  through  different  commands  or  in  different 
geographical  locations,  which  is  often  the  case  for  a  government-led  systems  integrator. 
The  development  of  subsystems  in  parallel  through  SBD  principles  lessens  the  need  to 
have  the  complete  design  team  centrally  located.  A  team  of  core  system  integrators,  as 
well  as  the  System  Chief  Engineer,  will  flow  the  subsystem  design  spaces  into  the 
analysis  tools  to  neck  down  the  solution  space  and  provide  feedback  to  the  team  on 
refinements  (Sobek  et  al.  1999). 

“Enabling  rapid  analysis  of  competing  systems  and  design  alternatives”  is  an 
advantage  if  the  right  analysis  tools  are  available.  This  may  not  yet  be  the  case  for  a 
majority  of  systems,  though  Naval  Surface  Warfare  Center  Carderock  Division  is 
working  on  a  few  for  ship  design  practices  (Gray  2015).  Assuming  these  tools  do  exist, 
there  is  the  ability  to  analyze  millions  of  solution  sets  to  support  shrinking  the  design 
space  at  a  rapid  rate. 

“Enabling  low  risk  and  high  reliability  designs  via  eliminating  infeasible  solutions 
first”  is  true  because  SBD  analyses  look  at  and  remove  infeasible  design.  Once  the 
solution  sets  narrow  enough,  modeling  and  simulation,  and  prototype  testing  occur.  This 
process  naturally  creates  more  robust  and  lower  risk  designs  because  instead  of  choosing 
a  less  effective  design  and  refining  it  through  iteration,  the  solution  becomes  apparent  by 
eliminating  surrounding  solution  sets. 

“Cost  commitments  deferred  until  sufficient  design  detail  permits  selection”  is  a 
major  advantage  this  entire  process.  As  the  system  understanding  matures  and  the 
infeasible  solutions  eliminated,  the  commitment  of  cost  delays  until  the  design  converges 
to  a  single  design  solution. 

The  majority  of  disadvantages  relate  to  the  lack  of  experience  and  guidance 
regarding  the  SBD  methodology  and  how  to  implement  SBD  applications  into  the  DOD 
acquisition  process.  There  needs  to  be  a  standard  process  to  follow  for  implementation 
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and  guidance  detailing  how  best  to  implement  SBD  as  not  only  a  process  but  within  the 
DOD  acquisition  framework;  standardization  is  required. 

“Higher  initial  costs  upfront  in  the  design  process”  and  “Commitment  of 
resources  needed  to  build  Assessment  Framework  and  perform  integration  of  SBD  tools” 
are  some  harsh  realities  of  SBD.  Resource  sponsors  will  have  to  consider  the  upfront 
costs  to  define  the  design  space,  generate  the  solution  sets,  determine  feasibility,  remove 
infeasible  solutions,  and  converge  on  a  set  of  more  feasible  solutions,  based  on  capability 
needs,  to  prove  out  viability  during  prototyping.  This  will  create  an  initial  work  bow 
wave  to  initiate  the  effort,  which  also  means  more  resources  and  funding.  The 
information  such  as  costs,  design  parameters,  risk,  fleet  value  and  technology  maturation, 
needs  to  be  concentrated  for  various  families  of  systems  to  perform  the  analysis.  The  data 
gathering  to  support  these  systems  is  burdensome  and  time  consuming.  The  integration  of 
the  database  and  analysis  tools  must  occur  for  the  AF,  which  is  an  analysis  within  itself, 
and  the  proper  structuring  is  crucial  because  it  is  the  weapon  for  defining  and  converging 
the  solution  sets. 

Prior  to  proceeding  with  SBD,  decision  makers  should  well  understand  the 
attributes  described  herein.  Although  beneficial,  the  methodology  does  have  its 
limitations,  but  the  promising  news  is  that  some  of  the  concerns  are  avoidable.  The  list 
above  helps  identify  the  generalities  among  the  case  studies  for  SBD,  as  defined  below. 

Using  SBD  should  be  faster.  The  first  three  case  studies  all  implemented  SBD  to 
get  to  a  solution  faster  mainly  due  to  schedule  constraints,  for  which  they  succeeded 
utilizing  this  methodology.  The  word  choice  “should”  was  thoughtfully  used  because,  as 
learned  from  the  LDUUV  case  study,  which  is  the  furthest  along  in  the  acquisition 
process,  the  approval  processes  and  oversight  required  via  the  Department  of  the  Navy 
Implementation  and  Operation  of  the  Defense  Acquisition  System  and  the  Joint 
Capabilities  Integration  and  Development  System  (SECNAVINST  5000. 2E)  caused 
schedule  delays.  The  DOD  acquisition  process  needs  to  be  analyzed  and  tailored  in  order 
to  take  advantage  of  the  positive  SBD  aspects. 
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Applying  SBD  methodically  shrinks  the  design  space  by  eliminating  infeasible 
solutions  in  a  step-like  process.  The  case  studies  all  went  through  steps  to  shrink  the 
design  space  after  discarding  infeasible  alternatives.  As  more  fidelity  develops,  the 
design  the  space  transforms.  Knowing  this,  should  design  space  reduction  milestones  be 
set  and  aligned  with  the  different  gates  or  milestones  within  the  SECNAVINST  5000. 2E? 
Some  technical  documents  may  permit  reductions  of  the  set  prior  to  adding  more  fidelity. 
There  is  a  balance  of  procedure,  stakeholder  input,  and  maintaining  a  rapid  prototyping 
methodology  to  consider  when  addressing  this  question. 

The  system  development  activity  can  leverage  SBD  in  different  capacities  and  at 
different  times  within  acquisition  framework.  The  LDUUV  and  the  Ship  to  Shore 
Connector  take  place  after  Milestone  A.  The  other  two  case  studies  are  feasibility  studies 
that  occurred  prior  to  Milestone  A.  Is  the  SBD  methodology  better  leveraged  in  a  study 
capacity  or  can  it  be  just  as  successful  implementing  it  through  Milestone  B  and  beyond? 
Should  there  be  a  best  practice  for  this?  When  analyzing  the  acquisition  framework,  we 
make  recommendations  for  implementing  SBD  as  well  as  how  to  tailor  the  Gate  criteria 
to  amplify  SBD’s  return  on  investment. 
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IV.  TAILORING  NAVY  ACQUISITION  FOR  SET  BASED  DESIGN 


Successful  application  of  SBD  in  defense  acquisition  depends  on  several  factors. 
This  chapter  explore  these  factors  and  how  they  may  influence  the  use  of  SBD  in 
program  development  efforts.  These  factors  include: 

1.  The  applicable  defense  acquisition  directive  and  instructions 

2.  Whether  the  system  being  developed  is  hardware  or  software  dominant 
and  the  acquisition  model  planned 

3.  The  SBD  tools  available  to  the  program, 

4.  If  additional  investment  in  tools  could  be  required  to  execute  SBD  within 
the  program, 

5.  Programmatic  factors,  such  as  the  effect  of  delaying  decisions  and 
possible  ways  to  communicate  in  sets 

6.  The  use  of  prototyping  in  SBD 

This  chapter  reviews  potential  acquisition  scenarios.  These  scenarios  provide  two 
examples  of  how  SBD  could  be  used  within  the  applicable  Navy’s  acquisition 
instructions,  and  to  highlight  the  process  tailoring  for  future  program  managers  to 
consider,  when  executing  SBD  acquisition  program.  The  review  begins  with  a  look  at 
applicable  instructions. 

A.  REVIEW  OF  APPLICABLE  DEFENSE  ACQUISITION  DOCUMENTS 

SBD  has  been  in  practice  in  industry  for  some  time.  However,  the  application  of 
SBD  methodology  in  design  is  new  to  DOD  acquisition  programs  and  projects.  Defense 
acquisition  policy  is  set  forth  in  Department  of  Defense  Directive  (DODD)  entitled  The 
Defense  Acquisition  System  (DODD  5000.01)  by  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics  (USD(AT&L)).  It  states  “the  primary  objective  of 
Defense  acquisition  is  to  acquire  quality  products  that  satisfy  user  needs  with  measurable 
improvements  to  mission  capability  and  operational  support,  in  a  timely  manner,  and  at  a 
fair  and  reasonable  price”  (2007,  3).  Additionally,  it  directs  acquisition  to  have  flexibility, 
responsiveness,  innovation,  discipline,  and  streamlined  and  effective  management.  Under 
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innovation,  it  charges  all  acquisition  professionals  to  “continuously  develop  and 
implement  initiatives  to  streamline  and  improve  the  Defense  Acquisition  System”  (2007, 
3).  USD(AT&L)  specifically  calls  on  Milestone  Decision  Authorities  (MDAs)  and 
Program  Managers  (PMs)  to  “examine  and,  as  appropriate,  adopt  innovative  practices 
(including  best  commercial  practices  and  electronic  business  solutions)  that  reduce  cycle 
time  and  cost,  and  encourage  teamwork”  (2007,  3).  The  study  of  SBD  to  improve  the 
design  processes  used  in  defense  acquisition  directly  supports  this  directive. 

The  incorporation  of  SBD  within  acquisition  programs  will  likely  require  tailoring 
of  existing  processes  and  procedures.  Operation  of  the  Defense  Acquisition  System  or 
DOD  Instruction  5000.02  (DODI  5000.02)  implements  the  DODD  5000.01  across  the 
department.  Like  the  DODD  5000.01,  the  DODI  5000.02  reiterates  the  authorization  for 
MDAs  to  tailor  programs  to  meet  the  DODD  5000.01  primary  objective.  DODI  5000.02 
specifically  authorizes  MDAs  to  tailor  regulatory  and  acquisition  procedures  as  long  as  it 
is  consistent  with  DODD  5000.01  (USD(AT&L)  2015).  Therefore,  any  tailoring  of 
processes,  reviews,  or  procedures,  to  incorporate  and  take  advantage  of  SBD,  is 
authorized  for  responsible  MDAs  as  they  see  appropriate. 

B.  FACTORS  TO  CONSIDER  BEFORE  SELECTING  TO  USE  SBD 

The  impact  of  utilizing  the  SBD  process  will  differ  depending  on  several  factors. 
This  section  identifies  those  factors  to  assist  both  the  program  manager  and  the  lead 
systems  engineer  in  determining  if  the  SBD  methodology  is  viable.  DODI  5000.02 
outlines  six  defense  acquisition  program  models  that  offer  baseline  examples  for  defense 
programs.  “Acquisition  programs  should  use  these  models  as  a  starting  point  in 
structuring  a  program  to  acquire  a  specific  product”  (USD(AT&L)  2015,  8).  Figure  16 
depicts  a  hardware  intensive  development  program  model,  which  is  the  typical  model  for 
most  DOD  acquisition  programs. 
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Figure  16.  Defense  Acquisition  Program  Model  -  Hardware  Intensive  Program. 

Source:  USD  (AT&L)  (2015,  9). 


The  program  model  contains  a  typical  system  life -cycle  familiar  to  any  systems 
engineering  effort:  requirements  and  product  definition  analysis,  risk  reduction, 
development,  testing,  production,  deployment,  and  sustainment  phases  punctuated  by 
major  investment  decisions  at  logical  programmatic  and  contractual  decision  points 
(USD(AT&L)  2015).  Each  decision  point  (Materiel  Development  Decision  (MDD),  CDD 
Validation,  Development  RFP  Release  Decision,  Full  Rate  Production  Decision, 
Milestone  A,  Milestone  B,  Milestone  C)  provides  the  MDA  with  the  ability  to  review 
both  the  programmatic  and  technical  status  and  risks  prior  to  proceeding  to  the  next 
acquisition  phase. 

Table  7  lists  the  six  DODI  5000.02  Defense  Program  Acquisition  Models.  The 
first  four  models  are  the  baseline  models,  while  Models  5  and  6  are  hybrid  models  for 
programs  that  have  hardware  and  software  intensive  programs  respectively.  Each  model 
contains  the  same  DOD  phases  and  decision  points,  but  the  software  models  add  software 
build  options  in  between  decision  points. 
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Table  7.  Defense  Program  Acquisition  Models. 
Source:  USD  (AT&L)  (2015,  9). 


Model  1 

Hardware  Intensive  Program 

Model  2 

Defense  Unique  Software  Intensive  Program 

Model  3 

Incrementally  Deployed  Software  Intensive  Program 

Model  4 

Accelerated  Acquisition  Program 

Model  5 

Hybrid  Program  A  (Hardware  Dominant) 

Model  6 

Hybrid  Program  B  (Software  Dominant) 

Figure  17  depicts  the  accelerated  acquisition  program  model.  In  this  model,  one 
can  see  that  milestones  A  and  B  are  put  together  resulting  in  one  preliminary  decision 
point  before  getting  to  the  milestones. 
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Materiel  Concurrent  Technology  Concurrent  Operations  &  Support 
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Legend:  /\m  Milestone  Decision  <^>  •  Decision  Point 


Figure  17.  Accelerated  Acquisition  Program  Model.  Source:  USD  (AT&L)  (2015, 

13). 


The  hardware  intensive  program  model  follows  the  traditional  DOD  acquisition 


approach  and  can  easily  incorporate  SBD.  The  chapter  expounds  upon  this  approach 
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later.  The  accelerated  acquisition  program  focuses  on  accelerating  the  program  to  deliver 
a  system  on  a  reduced  schedule.  The  accelerated  acquisition  program  model  combines 
Milestones  A  and  B  and  only  has  the  material  development  decision  point.  SBD 
complements  both  the  hardware  intensive  program  model  and  the  accelerated  acquisition 
program  model. 

SBD  may  be  an  approach  for  the  Accelerated  Acquisition  Program  model 
although  the  DOD  acquisition  implementation  may  require  tailoring  due  to  the  shortened 
acquisition  phases.  The  Accelerated  Acquisition  Model  reduces  the  number  of  decision 
points/Milestones  to  three:  the  MDD,  a  combined  Milestone  A/B,  and  Milestone  C. 

The  Defense  Unique  Software  Intensive  Program  and  the  Incrementally  Deployed 
Software  Intensive  Program  are  software  intensive  program  models,  while  Hybrid 
Program  B  (Software  Dominant)  is  a  software  dominant  hybrid  model.  Figure  18  shows 
an  example  of  an  incrementally  deployed  software  intensive  program  model. 
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Figure  18.  Incrementally  Deployed  Software  Intensive  Program.  Source:  USD 

(AT&L)(2015,  11). 


This  shows  how  the  acquisition  cycle  breaks  down  into  increments  focused  on 
software  capability.  This  does  not  fall  in  line  with  the  SBD  methodology  of  delaying 
decisions  until  feasibility  is  known  and  instead  focuses  on  an  incremental  approach  that 
builds  on  the  previous  version,  qualities  of  PBD.  Current  software  acquisitions  focus 
more  on  the  agile  development  process,  which  prioritizes  efforts  in  software  builds, 
versus  delaying  cost  commitments.  According  to  the  Project  Management  Institute,  agile 
software  development  focuses  on  quickly  producing  prototype  products  to  rapidly  solicit 
feedback  from  stakeholders,  “for  early,  measurable  (return  on  investment)  ROI  through 
defined,  iterative  delivery  of  product  increments”  (2016,  1).  Like  SBD,  these  software 
development  practices  emphasize  prototyping.  However,  agile  software  development 
progressively  elaborates  via  accelerated  PBD  increments,  not  by  maturing  solution  sets  as 
described  in  the  SBD  principles  (Project  Management  Institute  2016).  Perhaps  the  best 
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strategy  for  the  application  of  SBD  in  software  activities  is  to  apply  SBD  principles  to 
narrow  the  design  space  of  the  feasible  software  system  performance  and  functional 
requirements.  To  aid  in  the  elimination  of  infeasible  alternatives,  software  prototypes 
should  employ  agile  software  fast  incremental  cadences  to  obtain  rapid  feedback  from 
stakeholders.  Therefore,  the  use  of  SBD  in  software  intensive  systems  is  limited.  The 
focus  of  SBD  in  DOD  acquisition  should  be  on  hardware  intensive  systems  in  which 
requirements  ranges  and  tradeoff  curves  are  easily  defined  and  for  which  there  are  more 
defined  tools  available  for  analysis  among  solution  sets. 

When  determining  whether  to  employ  SBD  or  not,  one  must  consider  how  far 
along  the  system  development  is  and  at  what  point  is  the  program  in  the  acquisition  life 
cycle.  Programs  must  ensure  that  SBD  starts  early  on  when  design  trade  space  is  still 
available.  From  the  case  studies  reviewed  in  the  previous  chapter,  all  of  the  programs 
utilized  SBD  early  in  program  acquisition.  The  Ship  to  Shore  Connector  program  utilized 
SBD  during  preliminary  design  while  the  ACV  and  the  Small  Surface  Combatant  utilized 
SBD  for  an  AoA.  The  LDUUV  planned  to  use  SBD  through  the  preliminary  design  stage 
but  the  effort  has  not  begun  yet.  Conducting  SBD  late  in  the  program  life  cycle,  post 
Critical  Design  Review  (CDR),  will  not  provide  as  much  value  since  the  program  is 
already  committed  to  the  design,  thereby  preemptively  eliminating  the  number  of  feasible 
alternatives  before  leveraging  the  benefits  of  SBD. 

The  complexity  and  level  of  integration  in  the  program  are  also  important  to 
account  for  when  considering  SBD.  The  Ship  to  Shore  Connector,  Small  Surface 
Combatant,  ACV,  and  the  LDUUV  case  studies  are  all  programs  that  deliver  the  entire 
system,  providing  full  control  to  the  program  manager.  The  system  arrives  in  one  piece 
and  the  program  has  oversight  over  all  the  internal  interfaces.  SBD  might  not  work  as 
well  for  a  single  system  that  is  part  of  a  greater  system  of  systems  (SoS)  with  multiple 
programs  involved.  A  complex  SoS  with  multiple  interfaces  may  require  locking  down 
certain  specifications  earlier  in  the  acquisition  life  cycle  to  ensure  proper  integration, 
limiting  the  use  of  SBD.  For  example,  the  Preliminary  Design  Review  typically  results 
with  drafts  of  the  interface  control  documents,  finalized  by  the  CDR.  This  narrows  the 
trade  space,  limiting  the  value  SBD  may  provide.  The  practical  application  of  SBD  will 
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vary  among  subsystems  as  well.  In  Toyota’s  approach,  designers  decide  on  transmissions 
very  early  due  to  the  complexity  and  costs,  while  they  leave  open  decisions  on  exhaust 
system  designs  until  the  final  specification  (Ward  et  al.  1995). 

Hardware  specific  programs  that  are  early  in  the  life  cycle  leverage  SBD  the  best. 
Programs  with  few  complex  interfaces  or  well-defined  interfaces  are  preferred.  All  of 
these  factors  maximize  the  application  of  the  SBD  principles  to  get  the  full  benefit  from  a 
typical  point  based  design  approach. 

C.  SBD  TOOLS 

Proper  SBD  analyzes  variables  to  explore  trade-offs,  review  multiple  alternatives, 
look  for  intersections  of  sets,  and  narrow  the  sets  in  a  timely  manner  to  meet  design 
schedules.  In  order  to  conduct  this  analysis,  proper  tools  are  required  to  inject  and 
analyze  these  requirements  and  metrics.  Without  proper  tools  to  conduct  the  analysis,  the 
trade  space  cannot  be  analyzed  effectively.  Both  the  Small  Surface  Combatant  and  the 
ACV  utilized  a  DOORS  database  and  the  Framework  for  Assessing  Cost  and  Technology 
(FACT)  systems  engineering  toolsets.  The  teams  entered  cost  and  technical  parameters 
into  these  tools,  which  automate  calculations  for  the  various  studies  (Burrow  et  al.  n.d.). 
NAVSEA  has  developed  several  tools  to  analyze  complex  design  issues,  to  include  the 
Advanced  Ship  and  Submarine  Evaluation  Tool  (ASSET),  in  order  to  conduct  total  ship 
synthesis,  and  the  Leading  Edge  Architecture  for  Prototyping  Systems  (LEAPS),  to 
integrate  a  variety  of  analysis  tools  in  a  common  data  environment  (Kassel  2012). 
NAVSEA  continues  to  improve  incrementally  both  ASSET  and  LEAPS  in  order  to 
improve  their  ship  synthesis  capability.  NAVSEA  has  teamed  up  with  the  Office  of  Naval 
Research  (ONR),  the  DOD  High  Performance  Computers  Modernization  Office 
(HPCMO),  and  PEO  Ships  on  the  Computational  Research  and  Engineering  Acquisition 
Tools  and  Environments  (CREATE)  program  to  take  advantage  of  the  modern  increases 
in  computational  power  to  develop  these  toolsets  (Kassel  2012).  For  NAVSEA  programs 
such  as  the  LDUUV,  the  investment  in  these  tools  enables  the  program  office  to  conduct 
parallel  design  efforts  to  determine  the  intersections  of  the  design  sets.  Without  tools,  the 
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program  would  not  be  able  to  explore  fully  the  design  trade  space  while  still  meeting  the 
program  timelines. 

The  program  needs  to  make  a  determination  of  the  ability  to  tailor  the  tools  for  the 
program  prior  to  utilizing  a  SBD  approach.  The  tools  available  must  be  fully  capable  of 
taking  in  the  various  design  variables  and  conducting  the  proper  analysis  to  narrow  the 
design  sets.  If  tool  investment  is  required  for  SBD,  the  program  must  determine  the  return 
on  investment  of  these  tools.  A  program  will  likely  see  greater  reduction  in  cost  by 
pursuing  a  SBD  approach  vice  a  PBD  approach,  as  long  as  the  enabling  tools  are  readily 
available  and  the  program  can  leverage  the  past  investments  of  other  organizations.  If  the 
use  of  SBD  requires  the  creation  of  a  tool,  the  program  may  not  enjoy  the  cost  and 
schedule  benefits  over  a  point  based  design  method  due  to  the  time  and  money  needed  for 
tool  development.  If  the  program  needs  to  create  or  update  a  tool,  the  cost  and  schedule  to 
do  this  might  make  adopting  a  SBD  approach  less  cost-effective. 

D.  PROGRAMMATIC  FACTORS  WHEN  IMPLEMENTING  SBD 

Applying  SBD  to  a  program  will  incur  different  cost  and  schedule  risks  than  a 
typical  PBD  solution.  SBD  focuses  on  delaying  cost  commitments  until  there  is  sufficient 
knowledge  to  make  proper  decisions.  Figure  19  illustrates  the  effect  of  SBD  on  the 
design  process.  This  is  a  similar  figure  to  the  one  shown  in  Chapter  II. 
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Figure  19.  Impact  of  SBD  on  the  Design  Process.  Source:  Singer  et  al.  (2009,  8). 

The  figure  shows  the  percent  of  complete  development  versus  the  percent  of  costs 
committed.  In  a  PBD,  requirements  and  design  are  set  early  in  the  program  life  cycle. 
Program  costs  are  committed  as  a  result  of  making  requirements  and  design  decisions.  As 
costs  are  committed,  management  influence  decreases.  In  SBD,  programs  delay 
requirements  and  design  decisions  until  they  are  fully  understood.  By  delaying  decisions, 
program  managers  reduce  the  risk  of  committing  costs  to  the  wrong  solution  and 
maintain  the  ability  of  management  to  influence  the  solution  longer  into  the  development 
cycle,  thereby  increasing  the  ability  to  adjust  to  mid-development  requirements 
modifications.  Set  Based  Design  reduces  both  cost  and  schedule  risks  to  the  program  to 
ensure  that  they  design  and  deliver  the  right  product  (Singer  et  al.  2009).  Program 
managers  will  still  need  to  understand  how  implementing  SBD  will  change  the  cost  and 
schedule  of  a  program. 
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To  map  properly  the  design  space,  SBD  requires  increased  resources  early  in  the 
program  and  carries  multiple  solutions  forward,  unlike  the  traditional  PBD  approach.  A 
major  tenet  of  SBD,  SMEs  derive  multiple  sets  of  designs  in  parallel,  then  systematically 
narrow  them  by  identifying  solution  intersections.  In  the  ACV  case  study,  four  teams 
worked  in  parallel  to  analyze  requirements,  effectiveness,  trade  space,  and  affordability  to 
develop  alternatives  (Burrow  et  al.  n.d.).  The  Small  Surface  Combatant  conducted 
multiple  parallel  efforts  to  define  mission  areas,  the  requirement  trade  space,  and  the 
design  efforts  (Mebane  et  al.  2011).  While  the  parallel  efforts  reduced  the  total  schedule 
compared  to  a  linear  study,  requisite  resources  increased  to  support  multiple  teams. 

To  determine  if  SBD  can  be  employed  effectively,  time  and  manpower  must  be 
allocated  to  analyze  and  search  for  tools.  Funding  and  time  may  be  necessary  to  upgrade 
tools,  which  could  delay  the  start  of  the  SBD  process  to  conduct  tool  development. 
Fortunately,  in  the  area  of  ship  design,  NAVSEA  previously  invested  in  the  ASSET, 
LEAPS,  and  CREATE  tools.  While  NAVSEA  has  the  infrastructure  in  place,  the  tools 
must  be  configured  to  support  specific  programs,  and  upgraded  as  innovative,  more 
advanced  analysis  techniques  arise.  NAVSEA  took  advantage  of  investments  from  other 
programs  funded  by  ONR  and  HPCMO.  If  another  Systems  Command  (SYSCOM)  is 
unable  to  leverage  these  investments,  then  it  may  require  significant  investment  to 
conduct  SBD,  which  a  program  manager  may  not  have  in  his  or  her  budget.  Cost  and 
schedule  risks  increase  if  the  tool  development  does  not  mature  quickly  enough  to 
support  the  program  schedule. 

SBD  advocates  for  multiple  prototypes.  Prototyping  can  accelerate  knowledge 
gain  and  reduce  technical  risk.  It  also  increases  the  research  and  development  costs 
versus  baselines  with  minimal  or  no  prototyping  planned.  While  the  case  studies  analyzed 
were  requirements  focused,  Toyota  is  a  proponent  of  prototyping  and  including  suppliers 
in  the  SBD  process.  One  Toyota  exhaust  supplier  developed  approximately  10  to  20 
exhaust  prototypes  for  each  new  Toyota  car  design  (Ward  et  al.  1995).  This  was  key  in 
supporting  the  development  of  requirements  and  design  of  the  overall  system. 

Set  Based  Design  delays  design  decisions  until  the  feasibility  of  potential 

solutions  is  established,  enabling  trade-off  decisions  between  feasible  solutions.  Delaying 
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the  decision-making  until  later  in  the  program  life  cycle  means  that  a  majority  of  the 
decisions  happen  toward  the  latter  end  of  the  schedule.  Though  this  helps  improve 
confidence  in  the  design  decisions,  there  will  likely  be  less  slack  in  the  schedule,  causing 
problems  if  issues  arise.  Additionally,  there  is  a  risk  to  the  schedule  if  one  of  the  parallel 
efforts  falls  behind,  resulting  in  higher  cost  and  schedule  risks  for  the  dependent  systems, 
which  require  interface  with  the  system  under  development.  Often  in  the  DOD,  the 
maturity  of  interfaces  is  the  subject  of  review  at  various  technical  reviews  and  how 
detailed  they  appear  in  the  appropriate  interface  control  documents.  Delays  in  design 
decisions  may  also  delay  finalizing  these  documents,  causing  a  cascading  delay  to 
interfacing  systems  from  finalizing  their  designs.  Efforts  should  focus  on  defining 
interface  control  documents  early  enough  to  enable  interoperability  with  dependent 
systems,  however  define  them  to  be  flexible  enough  to  avoid  rework  once  the  design  is 
finalized  (e.g.,  providing  extra  discrete  signals  for  growth,  adopt  robust  protocol 
standards). 

Finally,  SBD  advocates  for  more  efficient  communication  among  the 
stakeholders.  There  must  be  a  consistent  approach  and  communication  plan  to  ensure  all 
stakeholder  expectations  maintain  alignment.  Toyota  was  able  to  reduce  both  the 
frequency  and  duration  of  communication  with  their  suppliers  by  employing  SBD  in 
parallel.  However,  Toyota  was  the  lead  organization  and  able  to  be  directive  to  their 
suppliers.  Employing  SBD  in  DOD  acquisition  will  require  support  from  all  levels  of 
DOD  organizations  involved,  though  it  is  unclear  if  SBD  can  improve  communication 
efficiency  within  the  DOD.  The  organizational  structure  of  stakeholders  in  many  complex 
DOD  systems  is  more  horizontal  than  the  top-down  Toyota-to-suppliers  model.  Most 
technical  communication  in  the  DOD  is  via  written  specifications.  The  development  of 
sets  of  detailed  specifications  for  multiple  alternatives  is  not  practical.  Toyota 
communicates  via  ranges  to  define  the  set  of  solutions  (Sobek  et  al.  1999).  The  DOD 
could  adopt  a  similar  strategy  of  increased  use  of  acceptable  ranges  in  order  to  enable  set- 
based  communications.  Additionally,  the  use  of  Model  Based  Systems  Engineering 
(MBSE)  as  a  SBD  tool  facilitates  set-based  communications. 
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The  advancement  of  MBSE  tools  could  improve  future  set -based 
communications.  Modern  MBSE  system  models  consist  of  interactive  databases  of 
system  requirements,  attributes,  and  relationships  (Vitech  Corporation  2011).  These 
models  enable  the  generation  of  multiple  views  of  system  solutions  to  facilitate  analysis 
(Vitech  Corporation  2011).  Potential  solutions  arise  through  the  generation  of  a  set  of 
models.  The  maturation  of  a  set  of  individual  potential  architectures  would  empower 
rigorous  trade-off  analysis  to  eliminate  less  desirable  architectures  and  narrow  the 
solution  set  as  advocated  in  SBD. 

E.  THE  USE  OF  PROTOTYPING  TO  SUPPORT  SBD  IN  ACQUISITION 

Rapid,  developmental,  operational,  low  fidelity,  high  fidelity,  competitive:  all 
represent  adjectives  the  DOD  uses  to  define  prototyping  strategies.  The  evolution  of 
prototyping  extends  beyond  buying  down  technical  risk;  it  reduces  programmatic  costs, 
increases  pace  of  development,  supports  maturing  industrial  technical  competencies,  and 
informs  decision-making  earlier  in  the  acquisition  process  (Hencke  2014).  As  stated  by 
Borowski,  “Prototyping  enables  better  acquisition  outcomes  by  improving  the  reliability 
of  available  information”  (2012,  1).  Prototyping  does  not  need  to  be  a  complex  pre- 
production  model;  it  can  come  in  all  forms  such  as  a  concept,  subsystem,  or  end  item.  A 
prototype  is  a  test  article,  “Paper  studies  estimate  a  technology’s  capabilities,  prototyping 
demonstrates  those  capabilities  through  testing.  Test  articles  are  designed,  constructed, 
and  tested  to  demonstrate  the  capabilities  of  some  technology  or  system”  (Borowski 
2012,  1).  Based  on  this  understanding  of  prototypes,  those  used  prior  to  Milestone  B 
should  be  repetitive  low-fidelity  prototypes  with  short  development  timelines  to  inform 
early  design  decision-making.  In  fact,  it  is  now  a  requirement  to  prototype  prior  to 
Milestone  B.  Per  SECNAVINST  5000. 2E,  which  “requires  that  the  acquisition  strategy 
(interpreted  to  mean  technology  development  strategy  for  the  Technology  Development 
(TD)  phase)  for  each  major  defense  acquisition  program  provide  for  competitive 
prototypes  before  Milestone  B  unless  the  MDA  waives  the  requirement”  (2011,  2-17).  It 
is  important  for  the  program  to  stay  within  budget  and  to  use  prototypes  advantageously, 
in  a  cost-effective  manner,  to  gain  understanding,  mature  high-risk  technologies,  and 

determine  the  intersections  of  feasible  solutions.  The  SBD  methodology  supports  low- 
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fidelity  (cheap)  prototyping  early  and  often,  for  a  better  understanding  of  the  design 
space,  while  integrating  at  the  intersections  of  solution  spaces  in  order  to  narrow  the  set 
of  solutions  (Ghosh  and  Seering  2014). 

As  described  by  Ghosh  and  Seering,  developing  low-fidelity  prototypes  to  support 
examination  of  requirements  is  one  of  the  seven  key  characteristics  of  SBD  (2014).  SBD 
promotes  the  use  of  low-fidelity  prototyping,  but  at  some  point,  more  fidelity  emerges 
into  the  test  articles  to  advance  the  design  to  a  manufacture-able  product. 

Figure  20  demonstrates  a  good  comparison  between  the  Technology  Readiness 
Levels  (TRLs),  acquisition  framework,  and  two  major  classifications  of  prototypes, 
developmental  and  operational  (Hencke  2014).  Low-fidelity  prototyping  falls  in  the 
developmental  category.  TRLs  1-4,  or  the  pre-concept  and  material  solution  phases,  are 
the  most  appropriate  phases  to  maximize  the  usage  of  low-fidelity  prototyping  during  the 
SBD  process.  The  pre-Milestone  A  activities  align  with  defining  the  design  space  and 
adding  fidelity  to  narrow  the  solution  sets,  or  trade  space,  within  the  domain  disciplines. 
Although  TRLs  5  and  6  are  still  considered  developmental  prototypes,  the  demarcation 
between  low-fidelity  and  high-fidelity  should  occur  between  Milestones  A  and  B,  during 
the  technology  development  phase. 
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Solution  Technology  Maturation  Engineering  &  a.  OT&E&  Sustainment 

Analysis  &  Risk  Reduction  Manufacturing  Development  Deployment  &  Disposal 

Developmental  Prototypes 

•  Demonstrate  feasibility  of  an  integrated  capability. 

•  Provide  evidence  of  overcoming  specific  technical  risk 
barriers. 

•  Develop  sufficiently  detailed  cost  data  to  enable  cost- 
capability  trades. 


Operational  Prototypes 

•  Demonstrate  military  utility  of  integrated  capability 
solutions. 

•  Demonstrate  robust  fabrication  processes. 

•  Demonstrate  performance  in  specific  operational 
environments. 

•  Define  form,  fit,  function  and  "ilities" — e.g., 
supportabiity. 

•  Enable  business  case  analyses. 


Figure  20.  Prototyping  TRLs  within  the  Acquisition  Framework.  Source:  Hencke 

(2014,  12). 
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For  this  analysis,  assume  the  PDR  occurs  prior  to  Milestone  B.  The  most 
important  output  of  the  PDR  is  the  system  allocated  baseline.  According  to  AcqNotes,  the 
allocated  baseline  is  the: 

Definition  of  the  configuration  items  making  up  a  system,  and  then  how 
system  function  and  performance  requirements  are  allocated  across  lower 
level  configuration  items.  It  includes  all  functional  and  interface 
characteristics  that  are  allocated  from  the  top  level  system  or  higher-level 
configuration  items,  derived  requirements,  interface  requirements  with 
other  configuration  items,  design  constraints,  and  the  verification  required 
to  demonstrate  the  traceability  and  achievement  of  specified  functional, 
performance,  and  interface  characteristics.  (AcqNotes  2016) 

There  will  be  little  trade  space  available  after  the  allocated  baseline  is  produced, 
meaning  there  will  be  minimal  trade  space  left  post-PDR.  Therefore,  to  obtain  that  level 
of  higher  fidelity,  more  robust  prototypes  must  be  introduced  into  the  technology 
development  phase  to  either  mature  the  critical  technology  or  mature  the  system  to 
demonstrate  affordability.  These  higher  fidelity  prototypes  will  drive  the  SBD  process 
and  narrow  the  trade  space  to  a  minimum.  These  prototypes  are  the  more  enhanced,  more 
capable  developmental  prototypes.  Developmental  Prototyping  has  a  tipping  point  of 
diminishing  returns;  it  will  become  uneconomical  not  to  drive  the  design  to  the  allocated 
product  baseline,  also  known  as  the  build-to  specifications,  which  is  the  output  of  the 
CDR  (AcqNotes  2016).  Not  all  risk  reduces  and  the  detailed  design  remains  on  a  strict 
schedule.  In  general,  post  PDR,  one  allocated  system  baseline  should  be  carried  forward 
to  the  begin  preforming  detailed  design  and  use  full-scale  prototypes  as  needed  to 
demonstrate  in-situ,  operational  performance.  Therefore,  prototyping,  especially  at  the 
low-fidelity  level,  is  a  key  driver  for  the  SBD  process  pre-Milestone  B  and  will  better 
promote  trade-offs  and  the  exploration  of  the  design  space. 

F.  SELECT  DEFENSE  ACQUISITION  STRATEGY  SCENARIOS 

Tailoring  of  processes  at  the  program  level  is  how  a  POR  incorporates  and 
maximizes  SBD  into  their  program.  Each  program  will  likely  have  unique  acquisition 
strategy  considerations  and  applicable  SYSCOM  processes.  Programs  tailor  the  service- 
issued  instructions  as  approved  by  their  PM’s  cognizant  MDA.  These  service-issued 
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instructions  provide  additional  detail  on  how  to  execute  the  DODI  5000.02  within  their 
service.  The  Navy  has  issued  guidance  in  the  Department  of  the  Navy  Implementation 
and  Operation  of  the  Defense  Acquisition  System  and  the  Joint  Capabilities  Integration 
and  Development  System  (SECNAVINST  5000. 2E)  instruction,  which  was  signed  on  1 
September  2011.  The  document  describes  an  overview  of  the  Navy’s  acquisition 
management  process  including  the  2-Pass/6-Gate  DON  Requirements  and  Acquisition 
Governance  Process,  illustrated  in  Figure  21. 
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Figure  21.  2-Pass/6-Gate  DON  Requirements  and  Acquisition  Governance  Process. 

Source:  ASN(RD&A)  (2011,  1-60). 


The  SECNAV  stated  goal  of  the  2-Pass/6-Gate  process  is  to  “ensure  alignment 
between  Service-generated  capability  requirements  and  systems  acquisition,  while 
improving  senior  leadership  decision-making  through  better  understanding  of  risks  and 
costs  throughout  a  program’s  entire  development  cycle”  (2011,  1-51).  The  2-Pass/6-Gate 
process  applies  to  all  pre -Major  Defense  Acquisition  Programs  (MDAPs),  all  MDAP 
Acquisition  Category  I  (ACAT  I)  programs,  all  pre-Major  Automated  Information 
System  (MAIS)  programs,  all  MAIS  ACAT  IA  programs,  and  ACAT  II  (2011).  The  2- 
Pass/6-Gate  process  has  two  major  phases:  Pass  1  and  Pass  2.  The  Chief  of  Naval 
Operations  (CNO)  and  the  Commandant  of  the  Marine  Corps  (CMC)  are  the  chairpersons 
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for  Pass  1  reviews.  Their  designees  lead  these  reviews  via  the  R3B  process.  The  R3B  is 
“the  Navy’s  3-  and  4-star  forum  for  reviewing  and  making  decisions  on  Navy 
requirements  and  resource  issues  (2011,  5).  The  SECNAV  instruction  establishes  Pass  1 
as  the  Navy’s  process  to  determine  capability  needs  and  encompasses  the  first  three  gates, 
focusing  on  documenting  high-level  requirements  identified  in  the  Joint  Capabilities 
Integration  and  Development  System  process  (2011).  According  to  SECNAVINST 
5000. 2E,  Pass  2  is  led  by  the  DON  component  acquisition  executive  (CAE),  the  Assistant 
Secretary  of  the  Navy  for  Research,  Development  and  Acquisition  (ASN(RD&A)),  and 
includes  Gates  4  through  the  Post-Integrate  Baseline  Review  Gate  6,  plus  all  follow-on 
Gate  6  reviews  (201 1).  The  process  elicits  and  documents  leadership  program  direction  at 
identified  decision  points  (2011).  The  difference  in  leadership  between  the  two  Passes  is 
significant  as  it  has  implications  on  the  level  of  design  executed  in  Pass  1  versus  Pass  2. 
A  successful  Pass  1  documents  the  capability  need  to  enable  the  acquisition  process 
executed  in  Pass  2  (2011). 

The  2-Pass/6-Gate  process  builds  on  procedures  developed  to  provide  oversight  of 
defense  acquisition  programs,  which  have  traditionally  been  PBD  style  programs.  Not 
until  recently  the  DOD  employed  SBD,  and  to  date,  only  in  a  handful  of  cases.  That  does 
not  mean  the  processes  are  not  effectively  tailorable  to  provide  oversight  to  SBD 
programs  as  well.  To  explore  potential  processes  tailoring,  consider  the  following  two 
acquisition  strategy  scenarios. 

1.  Scenario  1 

The  government  performs  SBD  from  MDD  until  sufficient  system  requirements 
and  system  performance  definitions  are  documented  in  a  System  Design  Specification 
(SDS)  as  it  is  referred  to  in  SECNAVINST  5000. 2E.  This  is  analogous  to  the  FDD  from 
the  SSC  case  study;  to  inform  a  REP  to  the  defense  industry  to  complete  the  detailed 
design  (Mebane  et  al.  2011). 

2.  Scenario  2 

The  government  performs  SBD  from  MDD  through  system  requirements 

definition,  detailed  design,  and  documents  the  system  design  in  a  Technical  Data  Package 
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(TDP)  to  enable  a  production  RFP  to  a  defense  industry  vendor  or  to  produce  Low  Rate 
Initial  Production  (LRIP)  items  for  testing  and  other  entrance  criteria  to  a  Milestone  C 
decision  (ASN(RD&A)  2011). 

Each  of  these  SBD  acquisition  scenarios  illustrates  potential  employment  of  SBD. 
The  two  scenarios  show  that  the  solution  space  narrows  to  different  levels  of  detail, 
depending  on  the  acquisition  strategy  and  the  handoff  system  development  at  different 
phases.  Figure  22  illustrates  how  the  SBD  solution  space  reduces  by  eliminating  the  least 
desirable  or  infeasible  solutions  as  more  knowledge  develops,  resulting  in  the  funneling 
effect  which  maps  the  broad  design  space  on  the  left  and  then  narrows  the  set  of  solutions 
as  the  program  executes  to  the  right,  ultimately  paring  down  to  the  final  solution. 
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Figure  22.  Solution  Space  Funnel  Scenarios.  Adapted  from  ASN(RD&A)  (2011,  1- 

60). 

G.  PASS  1  AND  GATE  4  IN  A  SET  BASED  DESIGN  DEVELOPMENT 

Due  to  the  nature  of  Pass  1,  the  two  scenarios  will  have  a  degree  of  commonality 
through  Gate  3  and  on  to  Gate  4  (first  gate  in  Pass  2).  Pass  1  establishes  and  approves 
high-level  capability  needs  and  transitions  them  to  the  CAE-led  Pass  2  for  the  TD  and 
Engineering  &  Manufacturing  Development  (EDM)  phases  (ASN(RD&A)  2011).  To 
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facilitate  this  transition  from  Pass  1  to  Pass  2,  the  team  recommends  specific  SBD 
tailoring  for  Pass  1  and  up  to  Gate  4.  It  is  assumed  that  the  selected  scenarios  will  diverge 
from  each  other  after  Milestone  A,  at  Gate  4,  as  they  proceed  through  the  Pass  2  process 
and  are  discussed  in  Sections  H  and  I.  The  following  is  an  examination  of  the  common 
tailoring  within  the  context  of  SBD,  which  applies  to  both  scenarios  introduced  in 
Section  F. 

1.  Gate  1  and  the  Materiel  Development  Decision  (MDD) 

Set  Based  Design  influences  the  content  contained  within  the  Gate  1  entrance  and 
exit  criteria.  One  entrance  criterion  for  Gate  1  is  a  completed  Service  review  of  the  AoA 
Guidance  (ASN(RD&A)  2011).  If  it  has  been  determined  that  a  SBD  approach  is 
desirable,  the  AoA  Guidance  should  document  it  here.  In  the  case  of  SBD,  the  Service 
review  of  the  AoA  Guidance  should  identify  the  boundaries  for  mapping  the  design 
space,  satisfying  the  capability  need  documented  in  the  ICD.  The  ACV  case  study 
presented  in  Chapter  III  is  a  good  example  determining  the  design  space  in  a  defense 
acquisition  program.  The  ACV  program  identified  the  “big  rocks”  (Burrow  et  al.  n.d.,  3). 
The  AoA  Guidance  would  initiate  the  SBD  process  by  identifying  the  “big  rocks”  as  the 
tradable  parameters,  and  more  importantly,  those  that  are  not  tradable  to  achieve  the 
desired  end-state.  At  this  point,  SBD  methodology  helps  to  explore  the  requirements 
space  and  evaluate  the  capability  gaps  with  the  Capability  Based  Assessment  (CBA).  To 
accomplish  this,  SBD  evaluates  Capability  Concepts  via  techniques  similar  to  the 
Capability  Concept  Bull’s-eye  chart  and  the  CCW,  described  in  Chapter  III. 

An  application  of  SBD  methodology  at  this  stage  will  inform  the  MDD.  The 
following  is  a  list  of  the  SBD  principles  presented  as  a  review  and  a  map  to  the  relevant 
applications  for  this  stage  (Sobek  et  al.  1999): 

2.  Map  the  Design  Space 

a.  Define  feasible  regions  -  Determine  all  feasible  concepts  that  will 
satisfy  the  capability  gap  identified  in  the  CBA. 
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b.  Explore  trade-off  by  designing  multiple  alternatives  -  Identify  the  “big 
rocks”  and  their  relative  importance  to  enable  trade  analysis  and 
Capability  Concept  scoring. 

c.  Communicate  sets  of  possibilities  -  Solicit  feedback  from 
Stakeholders  and  ultimately  the  selection  of  a  Capability  Concept  by 
the  MDA  at  MDD. 

3.  Integrate  by  Intersection 

a.  Look  for  the  intersection  of  feasible  sets  -  Mature  the  Capability 
Concepts  and  identify  their  intersections. 

b.  Impose  minimum  constraint  -  High-level  Capability  Concepts  only  at 
this  point. 

c.  Seek  conceptual  robustness  -  Determine  feasible  concepts  that  satisfy 
the  capability  gap  identified  in  the  CBA. 

4.  Establish  Feasibility  before  Commitment 

a.  Narrow  sets  gradually  while  increasing  detail  -  Executed  in  concert 
with  lb,  lc,  2a,  2c,  and  3c. 

b.  Stay  within  sets  once  committed  -  Aggressively  apply  configuration 
control  to  requirements  baselines.  Perform  analysis  to  ensure  all  “big 
rocks”  are  traced  to  Measures  of  Effectiveness  that  support  closing  the 
identified  Capability  Gap. 

c.  Control  by  managing  uncertainty  at  process  gates  -  It  is  our 
recommendation  that  the  above  analysis  should  be  performed  to  enable 
the  selection  of  a  Capability  Concept(s)  at  the  MDD. 

The  MDD  is  after  Gate  1  and  is  “a  review  that  is  the  formal  entry  point  into  the 

acquisition  process  and  is  mandatory  for  all  programs.  A  successful  MDD  may  approve 

entry  into  the  acquisition  management  system”  (Defense  Acquisition  University  (DAU) 

2016).  Per  SECNAVINST  5000. 2E  pre-ACAT  I  programs  cannot  combine  the  Gate  1 

and  MDD  events  (2011).  Two  different  authorities  chair  these  two  events;  for  Gate  1,  it  is 

the  CNO  or  CMC  as  appropriate,  while  the  MDD  is  chaired  by  ASN(RD&A)  (2011).  By 
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prohibiting  their  combination,  the  process  provides  the  opportunity  to  ensure  alignment 
between  these  offices  for  ACAT  1  programs. 

Gate  1  approves  the  AoA  Guidance.  This  the  logical  starting  place  for  the 
application  of  SBD.  The  AoA  Guidance  begins  the  SBD  process  to  Map  the  Design 
Space,  while  the  MDD  approves  the  AoA  Guidance  and  thereby  documents  the  directions 
to  execute  a  SBD  approach  for  system  development. 

The  use  of  SBD  in  defense  acquisition  is  new  and  standard  practices  are  still 
being  defined.  Therefore,  it  requires  careful  thought  to  inform  stakeholders  of  the  method 
of  execution.  Another  tailoring  recommendation  is  the  development  of  the  draft  Systems 
Engineering  Plan  (SEP)  early  in  the  process.  As  written  in  the  2-Pass/6-Gate  process,  the 
draft  SEP  is  an  entry  criterion  to  Gate  3  (ASN(RD&A)  2011).  However,  given  direction 
from  the  MDD  to  proceed  with  a  SBD  approach,  the  draft  SEP  could  and  should  start 
prior  to  Gate  2,  in  order  to  provide  a  basis  for  communicating  the  SBD  execution  to 
inform  stakeholders. 

5.  Gate  2 

Progressing  further  down  Pass  1,  we  examine  Gate  2.  It  is  evident  that  the 
Alternative  System  Review  (ASR),  an  entrance  criterion  for  Gate  2,  should  be  tailored  for 
SBD.  According  to  the  Defense  Acquisition  Guidebook,  the  ASR  facilitates 
communication  among  end  user  and  defense  acquisition  stakeholders  to  enable  the 
development  of  a  draft  performance  specification  (DOD  2013).  In  a  PBD  approach,  once 
the  AoA  analysis  is  complete,  the  ASR  serves  to  review  the  results  and  select  the 
preferred  alternative  (DOD  2013).  Additionally,  this  meets  the  “preferred  alternative 
identified,”  as  a  Gate  2  entrance  criteria  (ASN(RD&A)  2011,  1-61). 

For  a  program  employing  SBD,  the  output  of  the  ASR  is  the  preferred  Capability 
Concept  if  not  already  determined  at  MDD.  While  this  is  similar  to  “preferred  alternative 
identified,”  in  the  baseline  SECNAV  instruction,  this  tailoring  provides  recognition  that 
the  possible  sets  of  system  configuration  alternatives  remain  in  a  development  and 
evaluation  stage  (ASN(RD&A)  2011,  1-61).  In  the  methodology  discussed  above, 
mapping  the  design  space  of  Capability  Concepts  and  identifying  the  preferred  Capability 
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Concept,  if  not  already  been  completed,  occurs  in  preparation  for  the  ASR.  This  decision 
point  appears  in  Figure  23. 


Figure  23.  Flow  of  the  Selection  of  the  Capability  Concept. 

The  ASR  should  focus  on  the  big  rock  performance  parameters  that  drive 
operational  effectiveness.  The  output  of  the  ASR  is  the  performance  parameters  and  the 
understanding  of  their  relative  importance  in  order  to  evaluate  the  set  of  possible  system 
configuration  alternatives.  These  performance  parameters  enable  the  reduction  of  the 
number  of  sets  to  a  manageable  number. 

The  SSC  and  ACV  case  studies  both  utilized  SBD  methodologies  to  improve  their 
understanding  of  requirements.  In  the  SSC  program,  the  ICD,  AoA,  R3B  disposition, 
technical  data  from  the  legacy  system,  and  other  lessons  learned  formulate  the  FDD.  The 
FDD  is  a  “set  of  operational  requirements  and  derived  parameters  used  to  initiate  the 
design  effort”  (Mebane  et  al.  2011,  83).  In  the  ACV  study,  the  comparison  of  the 
requirements  occurred  via  trade-off  analyses  with  the  big  rocks.  Therefore,  the  output  of 
the  ASR  could  enable  the  development  of  the  FDD  Development  Plan  and  a  draft  FDD 
that  elaborates  performance  needs  of  the  system  to  meet  operational  capability  gaps 
stated  in  the  ICD.  Figure  24  depicts  a  diagram  of  the  SSC  Specification  Tree  and  traces 
these  source  documents  to  the  FDD.  The  big  rock  performance  parameters  facilitate 
trade-off  curve  development  to  reduce  the  number  of  sets  of  feasible  system 
configuration  alternatives  and  define  the  FDD.  The  draft  FDD  is  refined  in  parallel  with 
the  CONOPS  and  CDD. 
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Figure  24.  Ship  to  Shore  Connector  (SSC)  Requirements  to  Specification  Trace 

Source:  Mebane  et  al.  (2011,  83). 


6.  Gate  3 

Entrance  criteria  for  Gate  3  also  requires  tailoring.  Currently,  the  entrance  criteria 
includes  a  completed  service  review  of  the  CDD  and  CONOPS,  and  a  draft  SDS 
Development  Plan  (ASN(RD&A)  2011).  Keeping  with  the  development  of  a  FDD,  the 
FDD  Development  Plan  should  describe  how  the  subsystem  FRDs  would  be  matured 
(Mebane  et  al.  2011).  In  the  SSC  article,  Mebane  and  his  co-authors  describe  the  FRDs  as 
“evolving  set(s)  of  assumptions  and  potential  requirements  that  further  defined  the 
element  trade  space  and  ultimately  constrained  element- specific  requirements”  (2011, 
83).  Analysis  of  big  rock  performance  parameters  can  eliminate  the  infeasible  sets  of 
system  configuration  alternatives,  in  order  to  refine  the  list  of  assumptions  for  each 
subsystem  and  further  mature  the  FRDs.  The  SBD  principles  are  presented  once  again 
with  example  ways  the  methodologies  apply  to  the  development  process  going  into  Gate 
3  (Sobek  et  al.  1999): 
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7. 


Map  the  Design  Space 


a.  Define  feasible  regions  -  Determine  feasible  system  configuration 
alternatives  that  will  satisfy  the  FDD  performance,  that  is  the  set  of 
various  subsystem  alternatives  and  begin  documenting  subsystem 
performance  and  performance  trade-off  curves. 

b.  Explore  trade-off  by  designing  multiple  alternatives  -  Identify  derived 
subsystem  performance  ranges  based  on  the  Big  Rock  Performance 
Parameters  trade-off  analysis  and  their  impact  on  system  level 
performance  to  enable  trade  analysis  and  subsystem  alternative 
scoring. 

c.  Communicate  sets  of  possibilities  -  Solicit  feedback  from  stakeholders 
and  document  in  the  FDD  and  draft  FRDs. 

8.  Integrate  by  Intersection 

a.  Look  for  the  intersection  of  feasible  sets  -  Mature  the  draft  FRDs  and 
identify  the  intersections  of  feasible  sets,  eliminating  infeasible  sets. 

b.  Impose  minimum  constraint  -  Focus  on  the  identification  of  derived 
subsystem  performance  parameter  ranges  to  enable  future  trade-offs. 

c.  Seek  conceptual  robustness  -  Identify  system  configuration 
alternatives  that  are  not  conceptually  robust  for  possible  elimination 
from  the  set  of  considered  solutions. 

9.  Establish  Feasibility  before  Commitment 

a.  Narrow  sets  gradually  while  increasing  detail  -  Executed  in  concert 
with  lb,  lc,  2a,  2c,  and  3c. 

b.  Stay  within  sets  once  committed  -  Maintain  traceability  of  derived 
subsystem  performance  ranges  and  functional  allocations  to  Big  Rock 
Performance  Parameters  and  the  CDD.  No  new  system-level 
performance  thresholds,  focus  is  on  getting  the  CDD  approved.  Any 
new  user  desires  should  be  identified  and  prioritized  for  future 
increments. 
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c.  Control  by  managing  uncertainty  at  process  gates  -  The  application  of 
these  methodologies  should  result  in  the  final  set  of  FRDs,  which  are 
the  basis  of  the  SDS  to  be  approved  at  Gate  4. 

10.  Gate  4 

The  purpose  of  the  Gate  4  review  is  to  approve  the  SDS  and  assess  program 
affordability  (ASN(RD&A)  2011).  Major  entrance  criteria  include  the  approved 
CONOPS,  CDD,  and  SDS  signed  by  the  PM,  responsible  SYSCOM  Chief  of 
Engineering,  and  the  resource  sponsor  (ASN(RD&A)  2011). 

The  SDS  under  consideration  for  approval  at  Gate  4  consists  of  the  finalized  FDD 
and  FRDs.  To  arrive  at  this,  the  scope  of  materiel  solution  space  reduces  significantly, 
beginning  from  sets  of  Capability  Concepts  prior  to  MDD  and  followed  by  the  selection 
of  one  Capability  Concept  at  MDD.  That  concept  is  investigated  by  considering  sets  of 
system  configuration  alternatives  (variations  of  subsystem  configurations  to  meet  mission 
requirements  in  the  final  CDD  and  traced  to  system  level  performance  documented  in  the 
FDD).  The  CDD  and  CONOPS  are  approved  at  Gate  3  (ASN(RD&A)  2011).  As  the  set 
of  system  configuration  alternatives  reduces,  the  FRDs  become  more  concrete.  Then, 
based  on  the  work  of  Mebane  and  his  contemporaries,  “the  requirements  in  the  FDD  and 
FRDs  were  subsequently  mapped  to  their  respective  SWBS  area  to  become  the  draft 
specification  for  SSC”  (2011,  83).  The  draft  specification  is  the  draft  SDS  and  consists  of 
the  final  system-level  FDD  and  subsystem-level  FRDs. 

H.  ACQUISITION  STRATEGY  SCENARIO  1 

Scenario  1  -  The  government  performs  SBD  from  MDD  until  sufficient  system 

requirements  and  system  performance  definition  are  documented  in  a  SDS.  This  allows  a 

Gate  5  review  to  approve  a  RFP  to  defense  industry  to  complete  the  detailed  design, 

FRIP,  and  testing  to  inform  a  Milestone  C  decision.  The  natural  progression  of  the 

system  under  development,  after  the  approval  of  the  SDS,  is  the  development  of  the  PD 

(NAVAIR  2015).  The  PD  includes  the  Allocated  Baseline  (ABF)  and  obtains  approval  at 

Preliminary  Design  Review  (PDR),  (NAVAIR  2015).  It  is  important  to  understand  that 

the  SDS  does  not  include  the  complete  set  of  required  information  and  documentation  for 
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the  PDR  Systems  Engineering  Technical  Review  (SETR)  (NAVAIR  2015).  Additionally, 
a  combined  System  Requirements  Review  (SRR)  and  System  Functional  Review  (SFR) 
during  FDD  and  FRDs  development  is  prudent  to  ensure  no  programmatic  elements  are 
overlooked  and  all  logistical  elements  are  addressed  (NAVAIR  2015).  The  combination 
of  these  events  is  a  significant  tailoring,  however,  one  that  is  appropriate  given  the 
development  of  the  FDD  and  FRDs  accomplishing  the  system  and  functional 
requirements  derivation  that  are  the  focus  of  the  SRR  and  SFR  reviews.  The  combined 
SRR  and  SFR  represent  the  control  gate  to  select  the  preferred  subsystem  configuration 
of  the  possible  set  of  feasible  subsystem  configurations.  Figure  25  builds  on  flow  diagram 
introduced  in  Figure  23  to  illustrate  these  decision  points. 


Figure  25.  Flow  of  the  Selection  of  the  Subsystem  Configuration  through  SRR/ 

SFR. 


The  specifications  defined  in  the  FDD  are  allocated  to  the  respective  FRDs.  A 
draft  FRD  is  therefore  the  set  of  possible  subsystem  configuration  design  solutions  to 
meet  the  requirements  allocated  to  a  particular  subsystem.  The  tradeoff  analysis  and 
elimination  of  infeasible  alternatives  narrows  the  design  space  of  each  of  the  individual 
FRDs.  The  FRDs  mature  as  the  final  subsystem  solutions  emerge  and  the  final  FRDs  gain 
approval  at  the  PDR.  Figure  26  depicts  the  decision  flow  to  this  point. 
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Figure  26.  Flow  of  the  Selection  of  the  Subsystem  Configuration  through  PDR. 


The  completion  of  SETR-required  actions  and  documentation  is  required  in  both 
PBD  and  SBD  efforts.  Therefore,  there  exist  programmatic  and  logistical  items  that  must 
be  covered  as  part  of  the  SETR  process;  examples  include  revision  of  the  SEP,  Software 
Development  Plan,  the  Program  Protection  Plan  (NAVAIR  2015).  These  items  can 
require  significant  effort.  Typically,  they  performed  through  a  combination  of  program 
office  and  either  defense  industry,  Federally  Funded  Research  and  Development  Center 
(FFRDC),  or  government  engineering  organization  activities  (DAU  2016).  Set  Based 
Design  targeted  the  design  of  the  system  itself.  Many  of  these  programmatic  and 
logistical  items  may  be  DOD,  Service,  or  SYSCOM  specific  and  will  be  in  addition  to 
supporting  the  SBD  execution.  One  strategy  to  accomplish  tasks  outside  the  SBD 
activities  is  through  a  TD  contract,  delivery  order,  or  appropriate  task  order  to  perform 
the  ancillary  SETR  activities  in  parallel  to  SBD. 

The  TD  contract  could  be  an  umbrella  effort  for  which  the  SBD,  programmatic, 
and  logistical  items  are  all  elements.  In  addition  to  the  FRDs  developed  via  the  SBD 
methodology,  the  TD  effort  would  then  complete  the  other  ABL  specification 
documentation  not  covered  by  the  SDS  consisting  of  the  FDD  and  supporting  FRDs.  The 
TD  effort  would  then  support  all  engineering  efforts  through  PDR. 

If  leadership  so  decides,  the  ancillary  SETR  activities  could  be  accomplished  via 
a  discrete  effort  outside  of  the  SBD  effort.  If  so,  care  must  be  taken  when  constructing 
Statements  of  Work  (SOWs)  to  avoid  duplication  of  work.  Whether  a  multiple  effort 
strategy  or  singular  umbrella  TD  approach  is  selected,  it  is  important  to  acknowledge  and 
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plan  for  these  activities  that  are  necessary  for  a  successful  PDR  that  will  be  outside  of  the 
SBD  engineering  effort. 

Upon  completion  of  a  positive  PDR,  the  SBD  process  shifts  its  focus  from  ABL 
development  to  Product  Baseline  (PBL)  determination.  The  necessary  competitive  RFP 
preparations  are  completed  as  normal.  The  SOW  in  the  RFP  will  reference  the  ABL 
documents  resulting  from  the  SBD  activities.  For  the  purposes  of  this  scenario  study,  it  is 
assumed  that  the  SOW  requires  the  use  of  SBD  through  the  development  of  the  PBL  and 
approval  of  the  PBL  at  CDR.  Once  awarded,  the  EDM  contract  must  be  baselined. 

Upon  completion  of  the  ABL,  the  design  being  executed  under  the  EDM  contract 
shifts  from  finalizing  the  ABL  to  the  application  of  SBD  principles  to  potential 
component  configuration  solution  sets.  The  design  space  of  potential  PBL  solutions  must 
be  mapped.  The  feasible  region  is  bounded  by  the  approved  ABL.  Sets  under 
consideration  are  the  various  feasible  subsystem  component  configurations  that  could  be 
adopted  to  meet  the  requirements  specified  in  the  ABL.  Then  the  integration  by 
intersection  of  feasible  sets  narrows  the  possible  configuration  sets  (Sobek  et  al.  1999). 
These  repetitive  processes  narrow  the  set  of  possible  component  configurations  and 
solutions  until  the  CDR  reviews  and  approves  a  PBL. 

I.  ACQUISITION  STRATEGY  SCENARIO  2 

Lor  the  second  scenario,  as  described  earlier,  SBD  is  performed  for  a  government- 
led  design,  i.e.,  producing  a  TDP  for  a  vendor  to  build  to  print  and  support  Milestone  C. 
The  major  difference  between  this  scenario  and  the  previous  is  that  government  working 
capital  organizations  and/or  LLRDCs  execute  the  development  activities,  removing  the 
RFP  for  system  development. 

The  two  scenarios’  SBD  approaches  are  identical  up  through  Gate  4.  As 
previously  discussed,  during  Gate  3,  the  SDS  development  plan  is  replaced  by  the  FDD 
development  plan,  which  includes  how  the  FRDs  mature,  which  in  turn  carries  over  to 
Gate  4.  Gate  4  approves  the  SDS,  which  for  SBD  is  the  collection  of  the  FDD  and  FRDs 
as  the  exit  criteria  for  Gate  4.  Sobek  and  his  colleagues’  SBD  principles  (Map  the  Design 
Space,  Integrate  by  Intersection,  and  Establish  Leasibility  before  Commitment),  need  to 
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be  exercised  to  pare  down  the  solution  set  to  one  baseline  system  configuration  as 
described  in  detail  in  the  former  scenario.  This  iteration  accompanies  the  exit  criteria  of 
Gate  4  and  leads  to  the  PDR,  which  would  best  occur  prior  to  Gate  5.  As  discussed,  the 
customary  outcome  of  PDR  is  an  established  ABL  placed  under  configuration  control 
(AcqNotes  2016).  Gate  5  is  normally  the  RFP  approved  for  release  gate  (ASN(RD&A) 
2011).  With  this  being  a  govemment-led  design,  a  development  RFP  is  not  necessary,  but 
that  does  not  mean  there  is  no  Gate  5.  Instead,  Gate  5  aligns  with  Milestone  B  to  assess 
the  program  progress  and  to  approve  entry  into  the  EMD  phase.  Additionally,  while  the 
contracting  actions  may  be  significantly  reduced  by  not  developing  an  RFP,  and  all  of  the 
associated  business  clearance  to  meet  government-industry  contracting  requirements,  it 
should  be  noted  that  internal  Government  Task  Orders,  Funding  Documents,  and 
Statements  of  Work  or  Objectives  still  need  to  be  developed  to  authorize  and  fund  work 
to  be  completed. 

As  discussed  in  Scenario  1,  Public  Law  requires  AC  AT  I  programs  to  complete  a 
PDR  prior  to  Milestone  B.  Other  ACATs  have  the  opportunity  to  hold  the  PDR  before  or 
after  this  milestone,  though  the  PDR  should  be  held  as  soon  as  the  design  maturity 
allows.  Holding  PDR  prior  to  Milestone  B  should  be  the  goal,  regardless  of  ACAT,  to 
enable  better  understanding  and  maturity  for  the  MDA  to  consider. 

FFRDCs  and  similar  government-sponsored  research  activities  perform  systems 
engineering,  perform  analysis,  and  often  have  significant  laboratory  facilities 
(Department  of  Homeland  Security  2007).  The  government  (more  specifically  the  Navy) 
can  utilize  the  existing  Naval  Research  and  Development  Establishment  (NRDE)  for  the 
TD  phase  to  prepare  for  PDR.  The  NRDE  Working  Capital  Fund  model  ensures  the 
Warfare  Centers  remain  relevant  and  responsive  to  the  SYSCOMs  and  PEOs  (U.S.  Naval 
Research  Advisory  Committee  2010).  According  to  the  Federal  Acquisition  Regulation 
6.302-3  paragraph  (a)(2)(i),  there  is  no  requirement  for  full-and-open  competition  when 
the  provision  to  “establish  or  maintain  an  essential  engineering,  research,  or  development 
capability  to  be  provided  by  an  educational  or  other  nonprofit  institution  or  a  federally 
funded  research  and  development  center”  can  be  applied  (Federal  Acquisition  Regulation 
2016). 


87 


This  exemption  eliminates  the  need  to  release  an  RFP  when  contracting  with 
organizations  such  as  those,  which  make  up  the  NRDE.  Therefore,  leveraging  the 
facilities  and  workforce  within  the  NRDE  to  execute  the  TD  phase  of  the  SBD  scenario 
can  accelerate  the  acquisition  timeline  versus  Scenario  1.  Since  the  technical 
communities  of  the  NRDE  are  government  organizations,  intellectual  property  rights  that 
must  be  considered  when  dealing  across  industry  partners  is  not  an  issue.  This  reduces 
barriers  to  partnering  and  sharing  of  capabilities  among  organizations  in  the  NRDE.  The 
14  University  Affiliated  Research  Centers,  such  as  John  Hopkins  University  Applied 
Physics  Laboratory,  Pennsylvania  State  University  Applied  Research  Laboratory, 
University  of  Hawaii  at  Manoa  Applied  Research  Laboratory,  and  others  are  another 
source  for  technical  expertise  (AcqNotes  2016).  The  benefits  of  a  government-led  design 
and  the  NRDE  community,  make  it  an  efficient,  flexible,  and  cost-effective  network,  as 
these  organizations  operate  at  cost,  leverage  valuable  government  investments,  and  are 
focused  on  serving  the  advancement  of  technology  for  the  betterment  of  the  government 
and  society.  This  research  and  development  network  matures  technology  and  helps 
determine  solutions.  Low-fidelity  prototyping  efforts,  utilizing  the  NRDE  in-house 
capabilities  and  test  facilities  to  gain  knowledge  and  narrow  solution  sets,  aid  in  the 
technology  maturation.  This  enables  the  design  and  documentation  of  the  ABL  prior  to 
Gate  5. 

Gate  5  will  enter  with  the  approved  ABL  and  will  still  align  with  Milestone  B  per 
the  traditional  SECNAVINST  5000. 02E  acquisition  framework.  The  effort  to  resource 
and  complete  documentation  required  to  prepare  the  RFP  and  select  the  vendor  is 
immense  and  time  consuming  (USD(AT&L)  2015).  Therefore,  in  this  scenario,  the 
removal  of  the  RFP  allows  for  the  ability  to  shift  some  events  to  the  left  for  earlier 
execution. 

Like  Scenario  1,  what  occurs  between  Gate  5  and  Gate  6  in  terms  of  the  SBD 
application  is  more  trade  space  exploration  and  set  reduction  from  PDR  to  CDR.  After 
the  successful  completion  of  PDR,  the  designers  commence  detailed  design.  For  example, 
the  NAVAIR  SETR  process  specifies  that  at  PDR,  the  breadth  of  design  is  still  being 
evaluated,  validated,  and  verified,  creating  an  opportunity  for  continued  SBD  principles 
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at  a  lower  level  in  terms  of  subsystems  and  components.  The  CDR  approves  the  PBL, 
which  “allows  the  completion  of  the  design  to  the  component  level”  (NAVAIR  2015, 
39).  The  PBL  makes  up  the  specifications  needed  for  inclusion  in  the  build-to-print  TDP. 
At  this  point,  the  opportunity  for  SBD  has  diminished  as  the  trade  space  is  closed.  Once 
the  TDP  is  developed,  the  effort  moves  to  preparing  for  Test  Readiness  Review, 
Milestone  C,  LRIP,  and  ultimately  a  full  rate  production  decision. 

J.  SCENARIO  COMPARISON 

Each  respective  scenario  covers  the  spectrum  of  the  most-likely  acquisition 
strategies  a  program  manager  may  encounter,  but  is  not  all-inclusive  and  every  program 
must  tailor  these  recommendations  to  meet  the  needs  of  the  program.  The  objective  was 
to  demonstrate  that  SBD  can  be  executed  effectively  whether  through  one  design  team  or 
through  a  vendor  handoff.  In  fact,  both  scenarios  are  very  similar  in  expressing  how  the 
acquisition  strategy  is  tailorable  to  incorporate  the  SBD  principles  and  methodology;  both 
close  their  respective  solution  space  after  the  CDR.  The  major  differences,  which  will  be 
further  discussed  below,  are  buried  in  the  process  and  inherent  obstacles  of  each  scenario. 

Scenario  1  creates  difficulty  for  the  government  to  manage  SBD  performance  and 
oversight.  It  is  hard  to  measure  how  well  SBD  performs.  It  is  a  methodology  that  tailors 
specifically  to  each  program.  How  the  vendor  chooses  to  implement  it  is  their 
prerogative.  A  couple  key  principles  of  SBD  are  delaying  design  decisions  and  longer 
periods  for  stakeholder  influence.  While  the  vendor  is  maturing  the  design,  decisions 
need  close  management  and  documentation  in  the  REP  to  ensure  the  stakeholders  are  the 
focal  point  of  the  design  decisions,  not  the  vendor,  driven  by  profit  potential. 

For  Scenario  2,  there  is  no  SBD  transition  between  actors,  which  allows  for  a 
more  integrated  design  development,  especially  from  the  ABL  to  the  PBL.  The  design 
development  is  more  of  a  partnership  because  it  is  government-to-government.  That  in 
nature  promotes  more  stakeholder  influence  and  ownership.  There  is  also  the  ability  to 
leverage  the  capabilities  amongst  the  NRDE  community,  which  are  diverse,  cost 
effective,  and  easily  accessible. 
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Table  8  summarizes  the  comparison  between  the  two  scenarios.  In  this  table,  key 
characteristics  of  program  execution  demonstrate  the  amount  of  influence  each  scenario 
has  with  respect  to  the  characteristic.  The  government  ranks  the  influence  in  terms  of 
low,  moderate,  or  high. 


Table  8.  Scenario  Characteristic  Comparison  Table. 


Characteristics 

Scenario  1 

Scenario  2 

Process  Flexibility 

Moderate 

High 

Design  Control 

Moderate 

High 

Resource  Accessibility 

High 

Moderate 

Stakeholder  Influence 

Moderate 

High 

Execution  Efficiency 

Moderate 

High 

Competition 

Moderate 

Moderate 

Design  Risk 

Low 

High 

The  first  characteristic  is  Process  Flexibility;  this  refers  to  the  control  over  the 
SBD  process  and  its  implementation.  Scenario  1  is  moderate  because  the  government 
loses  control  of  the  SBD  execution  once  the  design  is  turned  over  to  the  vendor. 
However,  Scenario  2  the  influence  is  high  because  the  government  maintains  control  of 
the  design,  which  creates  a  partnership  between  the  program  office  and  the  lead  systems 
integrator. 

The  second  characteristic  is  Design  Control;  this  references  the  government’s 
ability  to  influence  the  design  during  synthesis.  Again,  SBD  promotes  delaying  design 
decisions  until  a  better  understanding  of  the  system  emerges,  which  creates  more 
stakeholder  involvement.  With  Scenario  1,  the  vendor  handoff  removes  some  of  that 
influence,  therefore  scoring  a  moderate  rating.  The  vendor  in  turn  would  be  making  some 
of  those  critical  design  decisions  in  a  vacuum  to  mature  the  design;  this  degrades  some  of 
the  SBD  advantage.  Scenario  2  scores  a  high  rating  because,  again,  this  is  government  to 
government,  creating  an  environment  of  partnering. 

The  third  characteristic  is  Resource  Accessibility;  this  is  the  ability  to  allocate 
resources  to  execute  the  program  in  an  efficient  manner.  Scenario  1  yields  a  high  rating. 
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Vendors  have  the  ability  to  control  staffing  levels  to  handle  workload  changes.  Scenario  2 
has  a  moderate  rating  because  staff  level  management  in  the  government  can  be  difficult. 
The  government  does  not  have  the  labor  flexibility  of  industry  and  could  have  to  reach 
across  multiple  platforms  or  organizations  to  staff  projects  appropriately. 

The  forth  characteristic  is  Stakeholder  Influence;  this  is  addressed  in  the  second 
characteristic  analysis  and  is  scored  similarly  between  the  two  cases.  The  stakeholder 
influence  does  decrease  with  a  vendor  handoff. 

The  fifth  characteristic  is  Execution  Efficiency;  this  refers  to  the  program 
efficiency  and  ultimately  the  timeframe  associated  with  delivering  the  product  to  the 
fleet.  Assuming  all  other  programmatic  factors  are  equal  (i.e.,  labor,  skills,  design  tools, 
risks),  the  opportunity  to  not  have  to  prepare  an  REP  and  award  a  contract  creates  a 
significant  amount  of  time  saving.  Therefore,  Scenario  1  has  a  moderate  ranking  with 
Scenario  2  receiving  a  high  ranking  because  of  contact  award  relief. 

The  sixth  characteristic  is  Competition;  this  refers  to  utilizing  competition  within 
the  program  to  lower  costs  and  mature  technology.  Both  Scenario  1  and  Scenario  2 
receive  a  moderate  rating  but  each  promotes  competition  differently.  With  Scenario  1,  the 
competition  is  during  the  proposal-soliciting  phase.  Scenario  2  promotes  competition  by 
utilizing  small  contracting  vehicles  throughout  the  design  cycle  to  reach  industry  for 
more  specific  purposes.  An  example  of  this  would  be  vendors  competing  for  the  battery 
design  of  an  unmanned  vehicle. 

The  final  characteristic  is  Design  Risk;  this  refers  to  the  risk  endured  by  the 
government  with  a  deficient  design.  Scenario  1  has  a  low  rating  because  the  vendor 
assumes  the  design  risk  and  is  under  contract  to  deliver  a  functioning  product.  For 
Scenario  2,  since  the  government  executes  the  design,  the  government  assumes  more  risk. 
Deficiencies  will  require  more  funding  to  correct  therefore  causing  a  high  rating  for  this 
particular  characteristic. 
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K.  IMPLEMENTATION  DIFFERENCES  FROM  TOYOTA 


The  SBD  Scenarios  above  provide  guidance  for  implementation  of  SBD  in  DOD 
acquisition.  These  scenarios  aim  to  meet  SBD  principles  while  meeting  the  intent  of  the 
various  acquisition  policies.  However,  there  are  distinct  differences  between  the 
recommendations  and  Toyota’s  original  concurrent  engineering  practices.  The  most 
noticeable  difference  is  in  the  use  of  prototyping.  While  both  Toyota  and  DOD  focused 
on  multiple  prototypes,  Toyota  “developed  prototypes  of  an  extraordinary  number  of 
different  designs  for  subsystems”  (Sobek  et  al.  1998,  48)  even  through  the  production 
phase.  In  the  DOD  guidance,  the  team  has  identified  low  fidelity  prototyping  strictly  to 
understand  the  design  space  and  not  look  at  production  benefits.  In  addition,  to  maximize 
the  benefit  of  SBD,  the  CDR  locks  down  the  design.  Toyota,  on  the  other  hand,  utilizes 
multiple  designs  even  during  production. 

While  the  SBD  implementations  are  similar  in  nature,  Toyota’s  implementation 
has  two  distinct  benefits.  The  first  is  the  relationship  with  its  vendors.  Toyota  is  able  to 
take  advantage  of  its  relationship  with  vendors  and  work  with  them  to  expand  the  design 
space.  The  vendors  participate  in  the  SBD  process  and  work  with  Toyota  in  prototyping. 
In  the  DOD  guidance,  prototyping  feeds  design  requirements  for  contracting.  The 
winning  vendor  does  not  participate  in  the  DOD  SBD  process  but  leverages  their  own 
SBD  approach  if  required  in  the  contract,  in  order  to  reach  a  converged  design.  The 
second  benefit  is  Toyota’s  ability  to  continue  prototyping  beyond  the  CDR  and  into 
production.  Toyota  is  willing  to  spend  more  money  on  the  prototyping  to  find  a 
successful  design  because  they  know  it  will  eventually  lead  to  profit  via  a  quicker  design 
to  production  phase  In  DOD  acquisition,  the  government  cannot  afford  such  lax  policies 
with  spending.  While  SBD  has  seen  some  benefit  as  shown  in  the  case  studies,  it  remains 
to  be  seen  if  the  DOD  can  reap  the  same  benefits  as  Toyota. 

L.  TAILORING  NAVY  ACQUISITION  FOR  SET  BASED  DESIGN 

SUMMARY 

This  chapter  introduced  the  DOD  and  Navy  acquisition  frameworks  to  detail  the 
administrative  structure  and  tailoring  opportunities  afforded  within  the  instructions. 
Defense  acquisition  directives  and  instructions,  which  affect  the  application  of  SBD, 

92 


were  identified.  SBD  is  discussed  concerning  the  core  Defense  Acquisition  Program 
Models  and  how  hardware  intensive  systems  lend  themselves  more  to  the  use  of  SBD  as 
opposed  to  software  intensive  systems.  The  particular  tools  needed  for  SBD  execution 
and  the  programmatic  impacts  of  SBD  were  discussed,  including  delaying  decisions  and 
ways  to  communicate  effectively  in  sets.  The  role  of  prototyping  in  SBD  and  the  use  of 
prototyping  to  gain  technical  knowledge  and  narrow  the  solution  space  were  also 
examined.  Following  the  discussion,  two  SBD  scenarios  were  presented  and  analyzed  to 
demonstrate  how  tailoring  SBD  can  maximize  its  benefits.  A  comparison  of  the  two 
scenarios  follows  the  analysis  to  portray  the  differences  of  each  circumstance. 
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V.  CONCLUSION 


A.  SUMMARY 

This  project  examined  the  potential  use  of  SBD  principles  and  methodologies  as  a 
part  of  Defense  Acquisition.  Since  weapon  system  acquisition  has  been  plagued  by  time, 
funding,  and  requirement  constraints,  the  benefits  of  SBD  provide  motivation  for  such  a 
study.  Further,  it  sought  to  provide  specific  guidance  for  DOD  program  managers  and 
systems  engineers  who  may  choose  to  employ  SBD.  This  project  answered  the  following 
research  questions: 

1.  What  is  SBD  and  how  can  it  benefit  defense  acquisition? 

SBD  is  a  system  engineering  design  methodology  which  is  centered  around  the 
concept  of  maintaining  an  expanded  design  space  to  include  all  design  possibilities  for  as 
long  as  practical,  while  delaying  critical  design  decisions  until  sufficient  information  is 
known  to  eliminate  alternatives  that  are  infeasible  (Sobek  et  al.  1999).  The  paper 
examined  how,  according  to  Sobek  and  his  coauthors,  SBD  advocates  systematically 
narrowing  the  set  of  possible  designs  while  imposing  minimum  constraint  (1999).  Based 
on  the  principles  set  forth  in  their  work,  SBD  looks  for  the  intersection  of  the  different 
required  system  functions  among  the  feasible  sets,  and  eliminates  those  alternatives 
outside  these  overlapping  regions  (1999).  According  to  Sobek  and  his  cowriters,  the 
principle  integrating  by  intersection  contrasts  from  the  PBD  methodology,  which 
optimizes  functions  in  a  compartmentalized  fashion  and  then  attempts  to  bring  the 
optimized  functions  together  in  an  integrated  solution  at  the  end,  resulting  in  optimized 
subsystems,  but  an  overall  suboptimal  system  design  (1999).  By  integrating  at  the 
intersections  (vice  optimized  silo),  a  better  overall  system  design  can  be  achieved  (1999). 

SBD  has  the  potential  to  benefit  defense  acquisition  programs.  However,  even 
though  Toyota’s  use  of  SBD  proved  to  be  highly  successful,  employing  it  as  part  of 
government  acquisition  is  a  different  process.  According  to  the  case  study  of  Naval 
Surface  Warfare  Center  Carderock  Division’s  three-month  design  exercise  presented  by 
Mackenna  and  his  co-authors,  which  compared  and  contrasted  the  application  of  PBD 
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and  SBD,  SBD  is  more  flexible  when  faced  with  requirements  changes  during  design 
execution  (2014).  Their  results  showed  how  SBD  enabled  the  Carderock  SBD  team  to 
absorb  imposed  requirements  updates  more  easily  than  the  PBD  team  who  performed 
more  design  rework  and  assume  additional  design  risk  to  maintain  schedule  (2014).  A 
defense  acquisition  program  execution  could  use  SBD  to  maintain  flexibility  in  the  face 
of  externally  imposed  requirements  changes. 

Furthermore,  through  the  case  study  analysis  in  Chapter  III,  the  following  are 
additional  benefits  of  using.  SBD  establishes  system  design  feasibility  before 
commitment  where  design  development  is  not  limited  to  a  single  solution.  Engineers 
develop  parallel  system-level  designs  and  delay  decisions  about  system  requirements 
until  the  understanding  of  trade-offs  is  sufficient.  The  method  continually  promotes 
design  discovery  while  allowing  flexibility  for  requirements  change.  This  method  defers 
cost  commitment  until  sufficient  design  detail  permits  selection,  allowing  for  a  longer 
period  for  stakeholder  influence  and  feedback. 

2.  What  factors  make  a  program  a  good  candidate  for  employing  a  SBD 
approach  in  defense  acquisition? 

Hardware  intensive  systems  are  good  candidates  for  employing  SBD.  The  DODI 
5000.02  lists  six  different  acquisition  program  models,  one  of  which  is  hardware 
intensive  programs.  Hardware  intensive  systems  lead  themselves  to  the  development  of 
tradeoff  curves  and  surfaces  to  allow  for  easier  application  of  tradeoff  analyses  among 
sets  of  possible  solutions,  as  advocated  in  SBD.  Therefore,  hardware  intensive  programs 
are  good  candidates  to  use  SBD. 

Programs  using  SBD  should  use  the  correct  tools  and  prototyping.  Set  Based 
Design  is  superior  when  done  with  the  correct  tools,  which  includes  incorporating 
prototyping  into  the  program  baseline.  Good  candidates  for  the  SBD  methodology 
include  those  programs  that  have  either  access  to  or  the  funding  available  to  develop  the 
design  analysis  tools  needed.  Programs  that  seek  to  utilize  SBD  should  perform  sufficient 
planning  and  budgeting  to  employ  SBD  enabling  tools  such  as  FACT,  LEAPS,  and 
ASSET  early  in  the  design  process.  In  addition,  prototyping  should  be  budgeted  into  the 
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program  prior  to  establishing  the  program  baseline  to  support  low  fidelity  prototyping  of 
multiple  alternatives  to  facilitate  set  reduction. 

Set  Based  Design  allows  for  delaying  of  system  life  cycle  cost  commitment. 
However,  the  team  predicts  that  additional  costs  arise  early  in  the  design  process  to 
utilize  the  analysis  tools  to  create  low  fidelity  prototypes  to  map  and  narrow  the  design 
space.  Low  fidelity  prototyping  is  a  critical  enabler  for  the  SBD  process  pre-Milestone  B 
and  will  better  promote  trade-offs  and  the  exploration  of  the  design  space.  Ensuring 
sufficient  funding  is  in  place  at  key  times  in  the  acquisition  process  ensures  the  proper 
steps  to  map  and  reduce  design  space  adequately. 

3.  What  effect  does  SBD  have  on  overall  system  costs  and  risks  in  support  of 
defense  acquisition?  Are  the  potential  benefits  worth  it? 

Set  Based  Design  focuses  on  delaying  cost  commitments  until  there  is  sufficient 
knowledge  to  make  proper  decisions  while  mapping  and  narrowing  the  design  space. 
When  delaying  cost  commitments,  management  and  team  members  have  a  longer 
duration  of  influence,  which  reduces  risks  to  the  program  (Singer  et  al.  2009).  Feedback 
flows  into  the  system  design,  as  more  information  about  design  requirements  become 
apparent  and  better  understood.  SBD  requires  increased  funding  earlier  in  the  program  to 
map  properly  the  design  space  versus  the  alternative  PBD  methodology.  Higher  upfront 
costs,  in  order  to  utilize  tools  and  create  multiple  low  fidelity  prototypes  to  achieve  a 
more  global  solution,  are  worth  it,  as  the  outcome  is  closer  to  meeting  user  requirements 
when  delivered  to  the  fleet,  resulting  in  lower  overall  cost  and  risk.  In  theory  and  in 
practice  in  the  commercial  world,  SBD  is  a  better  alternative  for  both  cost  and  risk. 
Unfortunately,  there  is  very  little  literature  available  on  implementation  of  SBD  in  DOD 
acquisition.  It  remains  unknown  if  SBD  can  bring  the  same  benefits  to  defense  system 
acquisitions  as  it  has  to  industry. 

4.  What  instructions  and  processes  would  have  to  be  tailored  or  revised  to 
facilitate  PORs  to  use  SBD  in  their  development  activities? 

The  incorporation  of  SBD  within  acquisition  programs  will  likely  require  tailoring 
of  existing  DOD  processes.  However,  SBD  can  be  accommodated  within  the  existing 
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instructions  without  revision.  DODI  5000.02  reiterates  the  authorization  for  MDAs  to 
tailor  programs  to  meet  the  DODD  5000.01  primary  objective  (USD(AT&L)  2015). 
Tailoring  of  processes,  reviews,  or  procedures  to  incorporate  and  take  advantage  of  SBD 
design  processes  is  available  to  the  MDA  as  they  see  appropriate  (USD(AT&L)  2015). 
SECNAVINST  5000. 2E  describes  an  overview  of  the  Navy’s  acquisition  management 
process  including  the  2-Pass/6-Gate  DON  Requirements  and  Acquisition  Governance 
Process  (2011).  The  application  of  this  process  will  need  to  be  tailored  to  best  incorporate 
SBD  into  the  development  Navy  systems.  The  two  scenarios  provided  in  Chapter  IV 
show  examples  of  tailoring  the  2-Pass/6-Gate  process  to  employ  better  SBD  within  the 
Navy’s  acquisition  process. 

B.  REVIEW  OF  TAILORED  SET  BASED  DESIGN  SCENARIOS 

Using  lessons  learned  from  the  SBD  DOD  case  studies,  attributes  and 
commonalities  were  examined  to  form  guidelines  for  SBD  implementation  within  the 
SECNAVINST  5000. 2E  2-Pass/6-Gate  process.  Two  different  acquisition  strategy 
scenarios  were  analyzed,  one  utilizing  an  RFP  to  award  the  design  to  a  vendor,  while  the 
other  being  a  government  led  design  effort.  Each  scenario  should  implement  the  SBD 
methodology  in  a  comparable  manner  with  the  tailoring  of  the  gate  entrance  and  exit 
criteria  to  promote  SBD  characteristics.  The  SBD  process  can  be  partitioned  into  two 
major  phases.  The  pre-Milestone  A  phase  consists  of  mapping  and  defining  the 
requirements  space.  The  post-Milestone  A  phase  consists  of  mapping  and  defining  the 
material  space  and  narrowing  the  solution  sets  to  the  best-valued  design.  The  CDR,  or 
product  baseline,  is  the  most  appropriate  place  to  determine  the  design  space.  At  this 
point  in  the  acquisition  process,  continuing  to  make  design  changes  (leaving  trade  space 
open),  may  be  costly  and  inefficient.  The  differences  of  each  SBD  scenario  emerge  when 
comparing  programmatic  characteristics  such  as  process  flexibility,  design  control, 
resource  accessibility,  stakeholder  influence,  execution  efficiency,  competition,  and 
design  risk  within  the  bounds  of  each  acquisition  strategy.  While  utilizing  SBD,  it  is  best 
executed  within  the  government  led  design  team  construct.  There  are  also  notable 
differences  with  how  industry,  such  as  Toyota,  leverages  SBD  versus  the  government’s 
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ability  to  execute  the  methodology.  These  constraints  are  inherent  of  the  nature  of  DOD 
acquisition  and  Toyota’s  profit  driven  business  model. 

C.  FURTHER  RESEARCH  AREAS 

A  detailed  analysis  of  applicable  SBD  processes  for  potential  SBD  programs  or 
platforms  would  be  useful.  Further  research  should  look  into  the  development  of  targeted 
acquisition  strategies  for  possible  programs,  followed  by  the  examination  of  specific 
SETR  checklists.  Growing  and  maturing  MBSE  tools  to  perform  SBD  analyses  and 
eliminate  alternatives  is  another  area  that  would  advance  SBD  in  the  world  of  systems 
engineering.  This  would  enable  more  fidelity  in  the  specific  deliverables  applicable  to 
SBD.  Developing  SBD  guidelines  for  more  program  acquisition  models  such  as  the 
accelerated  and  incrementally  deployed  models  would  help  future  program  managers. 
Additionally,  further  education  and  training  of  the  government  workforce  to  employ 
existing  tools  is  needed;  in  parallel,  standardization  of  SBD  tools  and  mechanisms  used 
ought  to  be  investigated. 

D.  CONCLUSION 

This  paper  provides  the  guidelines  and  assumptions  for  how  to  apply  the  SBD 
methodology  within  the  constructs  of  the  DOD  acquisition  framework.  Resources,  risks, 
and  programmatic  factors  are  evaluated  against  the  PBD  methodology.  These 
recommendations  are  just  the  first  steps  for  the  long-term  successful  use  of  SBD  within 
the  DOD.  The  initial  foundation  for  applying  SBD  to  DOD  acquisition  has  been  built 
with  a  clear  understanding  of  how  to  execute  its  core  principles  and  leverage  its  key 
characteristics  while  abiding  by  the  acquisition  instructions.  The  recommendations 
provided  in  this  paper  attempt  to  break  the  ground  of  incorporating  the  SBD  methodology 
within  the  DOD,  a  mammoth  endeavor. 


99 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


100 


LIST  OF  REFERENCES 


AcqNotes.  2016.  “Systems  Engineering.”  August  20.  Accessed  August  21. 

http://www.acqnotes.com/acqnote/careerfields/configuration-baselines. 

Assistant  Secretary  of  the  Navy  for  Research,  Development,  and  Acquisition 

(ASN(RD&A).  Department  of  the  Navy  Implementation  and  Operation  of  the 
Defense  Acquisition  System  and  the  Joint  Capabilities  Integration  and 
Development  System  (SECNAV  Instruction  5000. 2E).  Washington,  DC:  United 
States  Navy,  2011. 

Borowski,  Samuel  M.  2012.  “Defense  Acquisition  University.”  DoDLive.  Accessed 

August  17,  2016.  http://dau.dodlive.mil/2015/06/26/competitive-prototyping-in- 
the-department-of-defense-suggestions-for-a-better-approach/. 

Brookings  Institution.  2016.  “Uncharted  Seas:  Maritime  Strategy  for  a  New  Era  of  Naval 
Challenges.”  Washington,  D.C.:  Brookings  Institution. 

Buckley,  Michael  E.,  Walter  L.  Mebane,  Craig  M.  Carleson,  Chris  Dowd,  and  David  J. 
Singer.  2011.  “Set-Based  Design  and  the  Ship  to  Shore  Connector.”  Arlington 
County,  VA:  United  States  Navy. 

Burrow,  John,  Norbert  Doerry,  Mark  Earnesty,  Joe  Was,  Jim  Myers,  Jeff  Banko,  Jeff 

McConnell,  Joshua  Pepper,  and  Tracy  Tafolla.  N.d.  “Concept  Exploration  of  the 
Amphibious  Combat  Vehcile.”  United  States  Navy  and  Marine  Corps. 

Byers,  Rich.  2016.  “Set-Based  Design  (SBD).”  Panama  City:  Naval  Surface  Warfare 
Center  Panama  City  Detachment. 

Defense  Acquisition  University  (DAU).  2016.  Initial  Capabilities  Document  (ICD). 
DAU.  Accessed  August.  https://dap.dau.mil/acquipedia/Pages/ 

ArticleDetails .  aspx?aid=bd5 1  Obfa-c  1  e9-4f5 1  -bf6 1  -f5f6305c  1 1 47 

Department  of  Defense  (DOD).  2013.  Defense  Acquisition  Guidebook.  DOD.  Accessed 
August  14,  2016.  https://acc.dau.mil/ 
CommunityBrowser.aspx?id=289207&lang=en-US 

Department  of  Homeland  Security.  2007.  “Establishing  or  Contracting  with  Federally 
Funded  Research  and  Development  Centers  (FFRDCs)  and  National 
Laboratories.”  Accessed  October  14,  2016.  https://www.dhs.gov/xlibrary/assets/ 
foia/mgmt-directive-143-04-establishing-contracting-federally-funded-r-and-d- 
centers-national-labs  .pdf 

Evans,  J.H.  1959.  “Basic  Design  Concepts.”  Naval  Engineers  Journal  21. 


101 


Federal  Acquisition  Regulation  (FAR).  2016.  Federal  Acquisition  Regulation  System. 
Accessed  October  14,  2016.  http://farsite.hill.af.mil/reghtml/regs/far2afmcfars/ 
fardfars/far/06  .htm#P  1 39_ 1 9945 

Garner,  Matt,  Norbert  Doerry,  Adrian  MacKenna,  Frank  Pearce,  Chris  Bassler,  Shari 
Hannapel,  and  Peter  McCauley.  2015.  “Concept  Exploration  Methods  for  the 
Small  Surface  Combatant.”  United  States  Navy. 

Ghosh,  Sourobh,  and  Warren  Seering.  “Set-Based  Thinking  in  the  Engineering  Design 
Community  and  Beyond.”  Paper  presented  at  the  ASME  International  Design 
Engineering  Technical  Conferences  and  Computers  and  Information  in 
Engineering  Conference,  Buffalo,  NY,  August  2014. 

Gray,  Dr.  Alexander.  2015.  “Set-Based  Early  Stage  Ship  Design.”  West  Bethesda,  MD: 
Naval  Surface  Warfare  Center  Carderock  Division. 

Hardro,  Peter.  2016.  “LDUUV  Set  Based  Design.”  Newport,  RI:  United  States  Navy. 

Hencke,  Richard,  Capt.  2014.  “Prototyping:  Increasing  the  Pace  of  Innovation.”  Defense 
AT&L,  July:  10-14. 

Kennedy,  Brian,  Durward  Sobek,  and  Michael  Kennedy.  2013.  “Reducing  Rework  by 

Applying  Set-Based  Practices  Early  in  the  Systems  Engineering  Process.”  Systems 
Engineering  17  no.  3:  278-296. 

MacKenna,  Adrian,  Jeffery  Hough,  Alexander  Gray,  and  Peter  McCauley.  2014. 

“Engineered  Resilient  Systems  (ERS)  for  Ship  Design  &  Acquisition.”  West 
Bethesda,  MD:  Naval  Surface  Warfare  Center  Carderock  Division. 

Mebane,  Walter  L.,  Craig  M.  Carlson,  Chris  Dowd,  David  J.  Singer,  and  Michael  E. 
Buckley.  2011.  “Set-Based  Design  and  the  Ship  to  Shore  Connector.”  Naval 
Engineers  Journal.  American  Society  of  Naval  Engineers  3:  79-92. 

Naval  Air  Systems  Command  (NAVAIR).  Systems  Engineering  Technical  Review 

Process  (NAVAIR  Instruction  4355. 19E).  Patuxent  River,  MD:  United  States 
Navy,  2015. 

Neel,  Timothy  E.  1991.  Concurrent  Engineering:  A  New  Paradigm.  Carlisle  Barracks, 
PA:  U.S.  Army  War  College, 

Project  Management  Institute.  2016.  Project  Management  Institute  Agile  Practices. 
Accessed  August,  http://www.pmi.org/learning/featured-topics/agile. 

Raudberget,  Dag.  2010.  “Practical  Applications  of  Set-Based  Concurrent  Engineering  in 
Industry.”  Strojniskivestnik  -  Journal  of  Mechanical  Engineering.  Sweden: 
Jonkoping  University  56,  no.  11:  685-695. 


102 


Singer,  David,  Norbert  Doerry,  and  Michael  Buckley.  2009.  “What  Is  Set-Based 
Design?”  ASNE  Naval  Engineers  Journal  121,  no.  4:  31-43. 

Sobek,  Durward  K.,  Allen  C.  Ward,  and  Jeffery  K  Liker.  1999.  “Toyota’s  Principles  of 
Set-Based  Concurrent  Engineering.”  Cambridge,  MA:  MIT  Sloan  Management 
Review,  vol.  40,  no.  2:  67-83. 

U.S.  Government  Accountability  Office  (GAO).  2013.  “Where  Should  Reform  Aim 

Next?”  Statement  of  Paul  L.  Francis,  Testimony  before  the  Committee  on  Armed 
Services,  House  of  Representatives.  Washington,  DC:  GAO. 

U.S.  Naval  Research  Advisory  Committee.  2010.  “Status  and  Future  of  the  Naval  R&D 
Establishment.”  Accessed  October  2016.  http://www.nrac.navy.mil/docs/ 

20 1 0_S  ummer_Study_Report.pdf. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Fogistics  (USD  (AT&F)). 
2007.  The  Defense  Acquisition  System  (DOD  Directive  5000.01).  USD(AT&F), 
Washington,  DC:  Department  of  Defense. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Fogistics  (USD  (AT&F)). 
2015.  Operation  of  the  Defense  Acquisition  System  (DOD  Instruction  5000.02). 
Washington,  DC:  Department  of  Defense. 

Ward,  Allen,  Jeffrey  K.  Fiker,  John  J.  Cristiano,  and  Durward  K.  Sobek  II.  1995.  “The 

Second  Toyota  Paradox:  How  Delaying  Decisions  Can  Make  Better  Cars  Faster.” 
Cambridge,  MA:  MIT  Sloan  Management  Review,  vol.  36,  no.  3:46-61. 

Winner,  Robert  I.,  James  P.  Pennell,  Harold  E.  Bertrand,  and  Marko  M.  G.  Slusarczuk. 
1988.  “The  Role  of  Concurrent  Engineering  in  Weapons  System  Acquisition.” 
Alexendria,  V A:  Institute  for  Defense  Analyses. 


103 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


104 


INITIAL  DISTRIBUTION  LIST 


1.  Defense  Technical  Information  Center 
Ft.  Bel  voir,  Virginia 

2.  Dudley  Knox  Library 
Naval  Postgraduate  School 
Monterey,  California 


105 


